khu  b dZddlZddlmZddlmZmZmZmZm Z m Z m Z m Z ddl Z ddlmZddlmcmZddlmZmZmZmZddlmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,dd l-m.Z.dd l/m0Z0dd l1m2Z2m3Z3d gZ4Gd dejjZ6GddejjZ7GddejjZ8Gdd ejjZ9 dxdee:e jvfdejjde:dee:e jvffdZ< dydee:e jvfdejjde:de=dee:e jvff dZ>dzde:de=de9fdZ?d{de:dee:effd Z@d{de:dee:effd!ZAe2id"e@d#$d%e@d#d&d'd()d*e@d#eed&d'd(+d,e@d#eed-d'd(+d.e@d#d/d'd()d0e@d#d/d'd()d1e@d#d/d'd()d2e@d#d&d'3d4e@d#d&d'3d5e@d#d/d'3d6e@d#d/d'3d7e@d#d/d'3d8e@d#d/d'd(d9:d;e@d#d/d'd(d9:d<e@d#d/d'd(d9:d=e@d#d>d?e@d#d>id@e@d#d>dAe@d#d>dBe@d#d>dCe@d#dD>dEe@d#dD>dFe@d#dG>dHe@d#dI>dJe@d#d&d'dIKdLe@d#dD>dMe@d#dD>dNe@dOdPe@d#dQdRdSdSTdUe@d#dQdRdSdSTdVe@d#dQdR3dWe@d#dQdRdSdSTdXeAd#dYdDZd[eAd#d&dDZeAd#d/d\ZeAd#d/dZeAd#d/dZeAd#d/dZd]ZBe3dzde=de9fd^ZCe3dzde=de9fd_ZDe3dzde=de9fd`ZEe3dzde=de9fdaZFe3dzde=de9fdbZGe3dzde=de9fdcZHe3dzde=de9fddZIe3dzde=de9fdeZJe3dzde=de9fdfZKe3dzde=de9fdgZLe3dzde=de9fdhZMe3dzde=de9fdiZNe3dzde=de9fdjZOe3dzde=de9fdkZPe3dzde=de9fdlZQe3dzde=de9fdmZRe3dzde=de9fdnZSe3dzde=de9fdoZTe3dzde=de9fdpZUe3dzde=de9fdqZVe3dzde=de9fdrZWe3dzde=de9fdsZXe3dzde=de9fdtZYe3dzde=de9fduZZe3dzde=de9fdvZ[e3dzde=de9fdwZ\y)|a EVA EVA from https://github.com/baaivision/EVA , paper: https://arxiv.org/abs/2211.07636 @article{EVA, title={EVA: Exploring the Limits of Masked Visual Representation Learning at Scale}, author={Fang, Yuxin and Wang, Wen and Xie, Binhui and Sun, Quan and Wu, Ledell and Wang, Xinggang and Huang, Tiejun and Wang, Xinlong and Cao, Yue}, journal={arXiv preprint arXiv:2211.07636}, year={2022} } EVA-02: A Visual Representation for Neon Genesis - https://arxiv.org/abs/2303.11331 @article{EVA02, title={EVA-02: A Visual Representation for Neon Genesis}, author={Fang, Yuxin and Sun, Quan and Wang, Xinggang and Huang, Tiejun and Wang, Xinlong and Cao, Yue}, journal={arXiv preprint arXiv:2303.11331}, year={2023} } This file contains EVA & EVA02 model implementations evolved from BEiT, additional models in vision_transformer.py. Modifications by / Copyright 2023 Ross Wightman, original copyrights below N)partial)AnyCallableDictListOptionalSetTupleUnion)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STDOPENAI_CLIP_MEANOPENAI_CLIP_STD) PatchEmbedMlpGluMlpSwiGLU LayerNormDropPath PatchDropoutRotaryEmbeddingCatapply_rot_embed_catapply_keep_indices_nlc trunc_normal_resample_patch_embedresample_abs_pos_embedglobal_pool_nlc to_2tupleuse_fused_attn AttentionRopeAttentionPoolLatent)build_model_with_cfg)feature_take_indices) checkpoint)generate_default_cfgsregister_modelEvaceZdZUdZej j eed< dde de dededede d e d e d e e d e e d edeffd Z dde ejde ejfdZxZS) EvaAttentionzG EVA Attention with ROPE, no k-bias, and fused/unfused qkv options fused_attndim num_headsqkv_bias qkv_fusedqkv_bias_separatenum_prefix_tokens attn_drop proj_drop attn_head_dim norm_layerqk_norm scale_normc t|| s| r | Jd||_||z} | | } | |jz}| dz|_||_t |_||_|rtj||dzd|_ dx|_ x|_ |_ |rtjtj ||_|j%dtj |dtjtj ||_ndx|_x|_|_nstj||||_ tj||d|_ tj||||_ d|_ dx|_x|_|_| r| |j*ntj,|_| r| |j*ntj,|_tj2||_| r| |ntj,|_tj|||_tj2||_y) a, Args: dim: Input dimension of the token embeddings num_heads: Number of attention heads qkv_bias: Whether to add a bias term to the query, key, and value projections qkv_fused: Whether qkv projections are fused into one projection or separate qkv_bias_separate: Whether to apply bias to qkv as a separate addition or part of F.linear() call num_prefix_tokens: Number of reg/cls tokens at the beginning of the sequence that should not have position embeddings applied attn_drop: Dropout rate for attention weights proj_drop: Dropout rate for the output projection attn_head_dim: Dimension of each attention head (if None, computed as dim // num_heads) norm_layer: Normalization layer constructor to use for QK and scale normalization qk_norm: Enable normalization of query (Q) and key (K) vectors with norm_layer scale_norm: Enable normalization (scaling) of attention output with norm_layer NzzEvaAttention.__init__5s>  ) i+i i)")#  $$Hdnn,% !2(*!2 yyhl?DH6: :DK :$+  ll5;;x+@A $$Xu{{8/DQV$W ll5;;x+@A :>> >dkDK))CADK))C>DK))CADKDH6: :DK :$+ 3:j/  3:j/  I.,6Jx(BKKM IIh, I.rope attn_maskc |j\}}}|j|j|j|}ntj|j|j |j f}|jr|j|}||z }n,tj||jj|}|j||d|jdjddddd}|jd\} } } n|j|j|||jdj!dd} |j#|j|||jdj!dd} |j%|j|||jdj!dd} |j'| |j)| } } ||j*} tj| ddddd| ddft-| dddd| dddf|gd j/| } tj| ddddd| ddft-| dddd| dddf|gd j/| } |j0r=tj2| | | ||j4r|j6j8nd }n| |j:z} | | j!d dz} |F|j=tj>}| jA|ddddddftCd } | jEd } |j7| } | | z}|j!ddj|||}|jG|}|jI|}|jK|}|S)aForward pass for the attention module. Args: x: Input tensor of shape (batch_size, sequence_length, embedding_dim) rope: Rotary position embeddings tensor for position-aware attention attn_mask: Optional attention mask to apply during attention computation Returns: Tensor of shape (batch_size, sequence_length, embedding_dim) N)weightr:r9rr"r,)rY dropout_pz-inf)&shaperBrIrGcatr;rKr0Flinearr[reshaper-permuteunbindrC transposerDrErNrOr1rtype_asr+scaled_dot_product_attentiontrainingr2pr?tobool masked_fillfloatsoftmaxrQrRr3)rSxrXrYBNCrBr.qkvnptattns rVforwardzEvaAttention.forwardws' ''1a 88 {{"hhqk 99dkk4;; %LM))((1+C8OC((1TXX__8LC++aAt~~r:BB1aAqQCjjmGAq! A&&q!T^^R@JJ1aPA A&&q!T^^R@JJ1aPA A&&q!T^^R@JJ1aPA{{1~t{{1~1  ((C 1Q4C4]+-@1aq=AQSW-XY_`aiijklA 1Q4C4]+-@1aq=AQSW-XY_`aiijklA ??..1a#.2mm$..**A DJJA B++D$%LL4 ''1dD!3C)D(DeFmT<>$'DqA KK1  % %aA . IIaL IIaL NN1 rW) TTFr"r`r`NNFTNN)__name__ __module__ __qualname____doc__rGjitFinalrp__annotations__intrrrrr>Tensorr} __classcell__rUs@rVr*r*0s %% !"&+%&!!+/-1!#@/@/@/ @/  @/ $ @/ #@/@/@/$C=@/!*@/@/@/J,004 @5<<(@  - @rWr*c$eZdZddddddddddddejedfded ed ed ed ed edededede dededede ede de de ef"fd Z dde e jde e jfdZxZS)EvaBlockT@Fr"evar`Nr,r-r.r/ mlp_ratio swiglu_mlp scale_mlpscale_attn_innerr1 attn_typer3r2 drop_path init_values act_layerr5r4c t||||_| dk(rtnt}|||||| | | ||| |_|+t j|tj|znd|_ | dkDr t| nt j|_ |||_t||z}|rG|rt!|||r|nd| |_nEt%||dz|r|ndt j&d| |_nt)||||r|nd| |_|+t j|tj|znd|_| dkDrt| |_yt j|_y) a Initialize the EVA transformer block. Args: dim: Input dimension of the token embeddings num_heads: Number of attention heads qkv_bias: Whether to use bias terms in query, key, value projections qkv_fused: Whether to use a single projection for query, key, value mlp_ratio: Ratio of MLP hidden dimension to input dimension swiglu_mlp: Whether to use SwiGLU activation in the MLP scale_mlp: Whether to use normalization in the MLP scale_attn_inner: Whether to use normalization within the attention mechanism num_prefix_tokens: Number of tokens at the beginning of the sequence (class tokens, etc.) attn_type: Type of attention module to use ('eva' or 'rope') proj_drop: Dropout rate for projection layers attn_drop: Dropout rate for attention matrix drop_path: Stochastic depth rate init_values: Initial value for LayerScale, None = no LayerScale act_layer: Activation layer constructor norm_layer: Normalization layer constructor attn_head_dim: Dimension of each attention head (if None, computed as dim // num_heads) rX r-r.r/r1r2r3r4r5r7Nr` in_featureshidden_featuresr5dropr]Frrr5r gate_lastrrrrr5r)r=r>norm1r r*r|r@rFrGonesgamma_1rrM drop_path1norm2rrmlprSiLUrgamma_2 drop_path2)rSr,r-r.r/rrrrr1rr3r2rrrr5r4attn_clsrrUs rVr>zEvaBlock.__init__sgR _ $-$7=\ /'!'   GRF]r||K%**S/$ABcg 1:R(9-R[[]_ cIo. ! #$3-6zD" " #$3a$7-6zD gg#"  /#)2: DHGRF]r||K%**S/$ABcg 1:R(9-R[[]rWrXrYc |ji||j|j|j|||z}||j |j |j |z}|S||j|j|j|j|||zz}||j |j|j |j |zz}|SN)rXrY)rrr|rrrrrrSrtrXrYs rVr}zEvaBlock.forwards << DOODIIdjjm$R[I$\]]ADOODHHTZZ]$;<rGrr}rrs@rVrrs8 ""!$#%*%&"!!!+/"$''#,+/%YSYSYS YS  YS  YSYSYS#YS #YSYSYSYSYS"%YS  !YS"!#YS$$C=%YSvx 5RWR^R^I_rWrc$eZdZdZddddddddddddej ej dfd ed ed ed ed e de dedededede de de de e de de de ef"fd Z dde ejde ejfdZxZS)EvaBlockPostNormzF EVA block w/ post-norm and support for swiglu, MLP norm scale, ROPE. TrrFr"r`Nr,r-r.r/rrrrrr1r3r2rrrr5r4c 6t||dk(rtnt}|||||| | | |||  |_|||_| dkDr t | ntj|_ t||z}|rG|rt|||r|nd| |_ nEt||dz|r|ndtjd| |_ nt||||r|nd| |_ |||_| dkDrt | |_ytj|_y) a Initialize the post-norm EVA transformer block. Args: dim: Input dimension of the token embeddings num_heads: Number of attention heads qkv_bias: Whether to use bias terms in query, key, value projections qkv_fused: Whether to use a single projection for query, key, value mlp_ratio: Ratio of MLP hidden dimension to input dimension swiglu_mlp: Whether to use SwiGLU activation in the MLP scale_mlp: Whether to use normalization in the MLP scale_attn_inner: Whether to use normalization within the attention mechanism num_prefix_tokens: Number of tokens at the beginning of the sequence (class tokens, etc.) attn_type: Type of attention module to use ('eva' or 'rope') proj_drop: Dropout rate for projection layers attn_drop: Dropout rate for attention matrix drop_path: Stochastic depth rate init_values: Initial value for LayerScale, None = no LayerScale (NOTE: ignored for post-norm block) act_layer: Activation layer constructor norm_layer: Normalization layer constructor attn_head_dim: Dimension of each attention head (if None, computed as dim // num_heads) rXrr`Nrr]Frr)r=r>r r*r|rrr@rMrrrrrrrrr)rSr,r-r.r/rrrrrr1r3r2rrrr5r4rrrUs rVr>zEvaBlockPostNorm.__init__#sR $-$7=\ /'!'    _ 1:R(9-R[[]cIo. ! #$3-6zD" " #$3a$7-6zD gg#"  /#)2: DH _ 1:R(9-R[[]rWrXrYc ||j|j|j|||z}||j|j |j |z}|Sr)rrr|rrrrs rVr}zEvaBlockPostNorm.forward|sV  499QTY9+W XY Y  488A; 78 8rWr)rrrrr@rrrrprrrrrr>rGrr}rrs@rVrr!s?P ""!"$#%*%&!!!+/"$''#%<<+/%WSWSWS WS  WS  WSWSWSWS#WS #WSWSWSWS"%WS  !WS"!#WS$$C=%WSrx 5RWR^R^I_rWrcQeZdZdZddddddddd d d d d d d d d d d d d edd dd d d dd d ddddd d ddf'deeeeeffdeeeeeffdedededededede de de de de de d ed!e d"e d#e d$e d%e d&e d'e d(e e d)e d*ed+e d,e d-e d.ed/e d0e d1e e d2e e d3e ed4e e d5e d6e d7e eeeefefd8e fNfd9 Z dWd;ZdZej&j(dXd?e d:dfd@Zej&j(dYdAe d:eeeffdBZej&j(d:ej fdCZdZdede ed:dfdDZd:eej:e ej:ffdEZ d[dFej:dGe eeeefdHe dIe dJe dKedLe d:eeej:eej:eej:fffdMZ d\dGeeeefdNe dOe fdPZ!dZdFej:dQe ed:ej:fdRZ"dFej:d:ej:fdSZ#dYdFej:dTe d:ej:fdUZ$dFej:d:ej:fdVZ%xZ&S)]r(a! Eva Vision Transformer w/ Abs & Rotary Pos Embed This class implements the EVA and EVA02 models that were based on the BEiT ViT variant * EVA - abs pos embed, global avg pool * EVA02 - abs + rope pos embed, global avg pool, SwiGLU, scale Norm in MLP (ala normformer) r9avg TrFrr`NrijgMbP?img_size patch_sizein_chans num_classes global_pool embed_dimdepthr-r.r/rrrrr drop_rate pos_drop_ratepatch_drop_rateproj_drop_rateattn_drop_ratedrop_path_rater5r class_tokennum_reg_tokensuse_abs_pos_embuse_rot_pos_embrope_grid_offsetrope_grid_indexing use_post_normuse_pre_transformer_normuse_post_transformer_norm use_fc_normattn_pool_num_headsattn_pool_mlp_ratiodynamic_img_sizedynamic_img_padref_feat_shapehead_init_scalec(t t2||dvsJ||_||_|x|_x|_|_|rdnd|z|_|$|_d|_ |}(|!|!})n|dk(})| | }*n|) }*i}+|$r|+jtddtd|||||%| d |+|_ |jj},t|jd r|jj!n|}-|r*t#j$t'j(dd|nd|_|r*t#j$t'j(d||nd|_|xr|j,du|_|r7t#j$t'j(d|,|jz|nd|_t#j2| |_|dkDrt7||jd |_nd|_|rA|& t;|&nd}&t=||zd|$rdn|jj>|&|||_ nd|_ |(r||nt#jB|_"t'jFd||D.cgc]}.|.jI}/}.|rtJntL}0t#jNtQ|D1cgc]&}1|0||| | | | | |||j|||/|1||(c}1|_)tQ|D1cgc]}1td|1||-c}1|_*|*r||nt#jB|_+|dk(r:|"xs|}"|#xs| }#tY|j |"|#|t"jZ|_.nd|_.|)r||nt#jB|_/t#j2||_0|dkDrt#jb||nt#jB|_2|jg|jh|j0tk|j0d|j*tk|j*d|j,tk|j,d|jmto|jdt"jbrtk|jdjpd|jdjpjrju|'|jdjvjrju|'yycc}.wcc}1wcc}1w)a Initialize the EVA Vision Transformer model. Args: img_size: Input image size (single int for square, or tuple for rectangular) patch_size: Patch size to divide image into tokens (single int for square, or tuple) in_chans: Number of input image channels num_classes: Number of classes (output dim) for classification head (final projection), 0 for pass-through global_pool: Type of global pooling for final sequence ('avg', 'token', 'map', etc.) embed_dim: Embedding dimension for tokens depth: Number of transformer blocks num_heads: Number of attention heads qkv_bias: Enable bias for query, key, value projections qkv_fused: Use a single projection for query, key, value mlp_ratio: Ratio of mlp hidden dim to embedding dim swiglu_mlp: Use SwiGLU activation in MLP scale_mlp: Apply scaling normalization in MLP (normformer style) scale_attn_inner: Apply scaling normalization inside attention attn_type: Type of attention module to use drop_rate: Dropout rate after final projection and pooling pos_drop_rate: Dropout rate for positional embeddings patch_drop_rate: Rate of dropping patches during training proj_drop_rate: Dropout rate for projections attn_drop_rate: Dropout rate for attention drop_path_rate: Stochastic depth rate norm_layer: Normalization layer constructor init_values: Initial layer-scale values class_token: Use class token num_reg_tokens: Number of additional learnable 'register' tokens to add to the sequence use_abs_pos_emb: Use absolute (learned) positional embeddings use_rot_pos_emb: Use rotary position embeddings rope_grid_offset: Offset for rotary position embedding grid rope_grid_indexing: Indexing mode for rotary position embeddings ('ij' or 'xy') use_post_norm: Use post-norm transformer block type use_pre_transformer_norm: Use normalization layer before transformer blocks use_post_transformer_norm: Use normalization layer after transformer blocks use_fc_norm: Use normalization layer after pooling, before final classifier attn_pool_num_heads: Number of heads in attention pooling attn_pool_mlp_ratio: MLP ratio in attention pooling dynamic_img_size: Support dynamic image sizes in forward pass dynamic_img_pad: Apply dynamic padding for irregular image sizes ref_feat_shape: Reference feature shape for rotary position embedding scale head_init_scale: Initialization scale for classification head weights )ravgmaxmaxtokenmapr"rFNrNHWC)strict_img_size output_fmt)rrrrrr: feat_ratio)rnT)r1return_indices) in_pixels feat_shaper grid_offset grid_indexing)r,r-r.r/rrrrrr1r3r2rr5rblocks.)modulenum_chs reductionr)r-rr5r{Gz?std)rr num_featureshead_hidden_sizerr1rgrad_checkpointingupdatedictr patch_embed num_patcheshasattrrr@rFrGrH cls_token reg_token cls_embed pos_embedrPpos_dropr patch_droprr grid_sizerXrMnorm_prelinspaceitemrr ModuleListrangeblocks feature_inforQr!r attn_poolfc_norm head_droprAheadapply _init_weightsrfix_init_weight isinstancer[datamul_r:)3rSrrrrrrrr-r.r/rrrrrrrrrrrr5rrrrrrrrrrrrrrrrractivate_pre_normactivate_fc_normactivate_post_norm embed_argsrrrtdprblock_fnirUs3 rVr>z Eva.__init__sj JJJJ&&ENNND1DN'2!^!K 0"'5  "* *e3  $ 0!: %5!5     d5VL M% !+--   &&22 -4T5E5E|-TD   ' ' )ZdGRekk!Q &BCX\Tbekk!^Y&OPhl$?4)?Q` KK;)?)?? KMei  ]3 Q *"&"8"8#DO #DO :H:TY~6Z^N*Y&#349I9I9S9S-,0 DIDI1B 9-  !&>5!IJAqvvxJJ'4#(mm$5\%%#$# #!##%#!1#"&"8"8((a&%' %#$ (QVV[P\^KLD'! yA F^/AZ *bkkm % "5"B "5"B 0--%'' DN"DN0@z),bkkm I.9DqBIIi5bkkm  4%%& >> % $..c 2 >> % $..c 2 >> % $..c 2  dii + $))** 4 II   ! ! & & 7 IINN   $ $_ 5 ,gK%#&^s1T+4+T08T5returncd}t|jD]m\}}||jjjj |dz||j jjj |dzoy)z=Fix initialization weights by rescaling based on layer depth.cR|jtjd|zy)Ng@)div_mathsqrt)paramlayer_ids rVrescalez$Eva.fix_init_weight..rescale\s JJtyyx0 1rWr"N) enumeraterr|rRr[r rfc2)rSrrlayers rVr zEva.fix_init_weightZsi 2 )5 =OHe EJJOO**//A > EIIMM((--x!| < =rWmct|tjrNt|jd|j *tj j|j yyy)zbInitialize weights for Linear layers. Args: m: Module to initialize. rrN)r r@rArr[r:initzeros_)rSr#s rVr zEva._init_weightscsH a # !(( ,vv!qvv&" $rWcddh}|S)z(Parameters to exclude from weight decay.rrr)rSnwds rVno_weight_decayzEva.no_weight_decaynsK( rWenablec||_y)z)Enable or disable gradient checkpointing.N)r)rSr*s rVset_grad_checkpointingzEva.set_grad_checkpointingts #)rWcoarsec$tdddg}|S)z(Create layer groupings for optimization.z ^cls_token|pos_embed|patch_embed)z^blocks\.(\d+)N)z^norm)i)stemr)r)rSr-matchers rV group_matcherzEva.group_matcherys!4-/CD rWc|jSN)r)rSs rVget_classifierzEva.get_classifiers yyrWc||_|||_|dkDr&tj|j||_ytj |_y)zReset the classifier head. Args: num_classes: Number of output classes. global_pool: Global pooling type. Nr)rrr@rArrMr)rSrrs rVreset_classifierzEva.reset_classifiersF'  "*D >IAoBIIdnnk: SUS^S^S` rWc:|jr|j\}}}}|j<|jj}t |j||f||j }nd}|j|d|}|j|jj||fnd}n4|j}|j|jjnd}|jAtj|jj|jddd|fd}|||z}|jg} |j9| j|jj|jddd| j|jj|jdddtj| |gzd}|j!|}|j"%|j#|\}} || t%||| }||fS)N)new_sizeold_sizer1r\)rcrr"r_)rrcrrrrr1viewrX get_embedrrGrdexpandrappendrrr) rSrtruHWrwprev_grid_sizer rot_pos_embedto_cat keep_indicess rV _pos_embedzEva._pos_embeds  JAq!Q~~)!%!1!1!;!;2NNV+&*&<&<  ! q"a AAEAVDII//q!f/=\`MI59YY5JDII//1PTM >> % 4>>00RDaHaPA  I A >> %F~~) dnn33AGGAJBGH MM$..// BC D &A3,A.A MM!  ?? &"ooa0OA|(\-E 6q- V -rWrtindicesreturn_prefix_tokensrQ stop_earlyrintermediates_onlyc |dvsJd|dk(}g} tt|j|\} } |j\} } }}|j |}|j |\}}|j |}tjjs|s |j}n|jd| dz}t|D]q\}}|jr-tjjst|||}n |||}|| vsN| j|r|j|n|s|jrD| Dcgc]}|ddd|jf}}| Dcgc]}|dd|jdf} }|ra|jj!||f\}}| Dcgc]6}|j#| ||dj%dd dd j'8} }tjjs|rt)t+| } |r| S|j|}|| fScc}wcc}wcc}w) a) Forward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if an int, if is a sequence, select by matching indices return_prefix_tokens: Return both prefix and spatial intermediate tokens norm: Apply norm layer to all intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features )NCHWNLCz>Output format for EVA-ViT features must be one of NCHW or NLC.rJNr"rXrr\r9r])r$lenrrcrrDrrGr is_scriptingr rr%r=rQr1dynamic_feat_sizergrh contiguouslistzip)rSrtrErFrQrGrrHrg intermediates take_indices max_indexru_heightwidthrArrblky prefix_tokensr>r?s rVforward_intermediateszEva.forward_intermediatess.(_,n.nn,& "6s4;;7G"Q i gg1fe   Q ??1-= MM!  99 ! ! #:[[F[[)a-0F' BFAs&&uyy/E/E/GsAM: .L $$TTYYq\qA  B  ! !ERSQq!D$:$:"::;SMSDQRqQq$"8"8"99:RMR ##55vuoFDAq^klYZQYYq!Q3;;Aq!QGRRTlMlyy%%',@ ]M!BCM  IIaL-TRms H;,I0;I prune_norm prune_headctt|j|\}}|jd|dz|_|rtj|_|r2d|_tj|_|jdd|S)z@ Prune layers not required for specified intermediates. Nr"rr) r$rMrr@rMrQrrr6)rSrEr]r^rTrUs rVprune_intermediate_layerszEva.prune_intermediate_layerssq#7s4;;7G"Q ikk.9q=1  DI !DN;;=DL  ! !!R (rW pool_typec|j|j|}|S| |jn|}t|||j}|S)N)rar1)rrrr1)rSrtras rVpoolzEva.pool sJ >> %q!AH(1(9D$$y AdF\F\ ]rWc<|j|}|j|\}}|j|}|jD]E}|jr-t j jst|||}<|||}G|j|}|S)zForward pass through feature extraction layers. Args: x: Input tensor. Returns: Feature tensor. rL) rrDrrrrGrrNr%rQ)rSrtrArYs rVforward_featureszEva.forward_featuress   Q ??1-= MM! ;; /C&&uyy/E/E/GsAM: .  / IIaLrW pre_logitsc|j|}|j|}|j|}|r|S|j|S)zForward pass through classifier head. Args: x: Feature tensor. pre_logits: Return pre-logits if True. Returns: Output tensor. )rcrrr)rSrtrfs rV forward_headzEva.forward_head'sA IIaL LLO NN1 q0DIIaL0rWcJ|j|}|j|}|S)zoForward pass. Args: x: Input tensor. Returns: Output tensor. )rerh)rSrts rVr}z Eva.forward6s)  ! !! $   a rW)rN)TFr3)NFFFrJF)r"FT)'rrrrrr rr rrprrrrr>r r@Moduler rGrignorer r)r,rrr1r4r6rrDrr\r`rcrerhr}rrs@rVr(r(s5868#$ !"!$#%*"!#%%'$&$&$&#,+/ $"#$($)&(&*"'-28<*.1537%*$)DH%*QN6CsCx01N6c5c?23N6 N6  N6  N6N6N6N6N6N6N6N6N6#N6 !N6"#N6$!%N6&#'N6(")N6*"+N6,"-N6.!/N60"%1N623N64 5N66"7N68"9N6:$;N6<!$=N6> ?N6@'+AN6B(0~CN6D"$EN6F"*#GN6H"*%IN6J#KN6L"MN6N%U5c?C+?%@AON6P#QN6`= 'ryy 'T ' YYS  YY)T)T)) YYDT#s(^ YY  aC ahsm aW[ a' uU\\8ELL3I%IJ' X8<).$$',< ||< eCcN34< #' <  <  < < !%<  tELL!5tELL7I)I#JJ K< @./$# 3S >* $ellx} %,,5<<( 1ell 1 1 1  %,, rW state_dictmodelprefixrc|jd|}|jDcic]\}}|jdd|}}}i}gd}t|}|jD]n\}}|r|j |s||d}|D]}|j|d|d}|j dr|jd d}|jd d }|jd d }|j dr|j ddz} |j dr|d| |d<|| d|d<n!|j dr|d| |d<|| d|d<|dk(r:d}|jdd}tj|j d|d<n>|dk(r#d}|jdjd}n|dk(r|jd}|||<q|Scc}}w)zConvert Perception Encoder weights. Args: state_dict: State dictionary to convert. model: Target model instance. prefix: Prefix to strip from keys. Returns: Converted state dictionary. rnzmodule.r) )conv1patch_embed.proj)positional_embeddingr)ztransformer.resblocks.r)ln_prer)ln_postrQ)ln_rQ)z ls_1.gammar)z ls_2.gammar)in_proj_zqkv.)out_projrR)zmlp.c_fcmlp.fc1)z mlp.c_projmlp.fc2Nrr"rzattn_pool.attnzattn_pool.layernormzattn_pool.normzattn_pool.probezattn_pool.latentz attn_pool.qkvr9r[zattn_pool.q.weightzattn_pool.kv.weightr:zattn_pool.q.biaszattn_pool.kv.biasrRz head.weightz head.biasclass_embeddingrr) getitemsreplacerM startswithrcendswithrjrGrH unsqueeze) rmrnroryrzout_dictswaps len_prefixspr,s rV _convert_perDs4J:D:J:J:LM$!Q!))Ir*A-MJMH EVJ  "1 <<'*+A (B "Q%A'A ( << $ *K8A /1ABA +-?@A||O,ggajAo::h'56tWH1267gH23ZZ'34Tc7H/045cdGH01 &[A Aq!A$)KK $;H[ ! # #A A((+A +  AA ?B OgNsG interpolation antialiasc i}|jd|}|jd|}|jd|}|jd|}d|vr t||Sd|vrt||dSd |vrd }n d |vrd }nd}|d z|v}|dz|v}t|}|jD]\} } |r| j |s| |d} d| vr%d| vrf|j j jj\} } } } | jd| k7s| jd| k7rt| | | f||d} no| dk(rj| jd|jjdk7rAt|ddrdn t|dd}t| |j j|||d} | jdd} | jdd } | jd!d"} | jd#d$} | jd%d&} | jd'd(} |r$| jd)d*} | jd+d,} |r#| d-vr| d.k(s| d/k(r| jd0d1} n| || <|S)2aZConvert patch embedding weight from manual patchify + linear proj to conv. Args: state_dict: Checkpoint state dictionary. model: Target model instance. interpolation: Interpolation method for resizing. antialias: Whether to use antialiasing when resizing. Returns: Filtered state dictionary. model_emarnrrmzvisual.conv1.weightz conv1.weightr)rozvisual.trunk.pos_embedz visual.trunk.zvisual.pos_embedvisual. mask_tokenzblocks.0.attn.q_proj.weightNrXzpatch_embed.proj.weightr\rbT)rrverboserr"no_embed_classFrr1)r8r1rrrz mlp.ffn_lnzmlp.normzattn.inner_attn_lnz attn.normzmlp.w12ryzmlp.w1z mlp.fc1_gzmlp.w2z mlp.fc1_xzmlp.w3rzrIz q_proj.biasrKz v_proj.bias)rzlm_head.weightz lm_head.bias norm.weight norm.biasrrrQr)r|rrMr}rrrRr[rcrrgetattrrrr~)rmrnrrrro mim_weightsno_qkvrryrzrVr>r?r1s rVcheckpoint_filter_fnrs"H Z8J4J*5J j9J *:u-- : %:uR88 :-  z )<':5K 3 3z AFVJ  "21 <<'*+A Q;  $ )**//66<Fr"r rrrrr-r/r.rrrrrrr)$vit_medium_patch16_rope_reg1_gap_256rrs rVrrsYJ  t: tY]^hYslrYs tE LrWc dtdddddddddd ddd }td d |it|fi|}|S)Nrrrr~TFrr"rrr)%vit_mediumd_patch16_rope_reg1_gap_256rrs rVrr*sYJ  uJ uZ^_iZtmsZt uE LrWc dtdddddddddd ddd }td d |it|fi|}|S)Nrrir TrFr^rrr)%vit_betwixt_patch16_rope_reg4_gap_256rrs rVrr?sYJ  uJ uZ^_iZtmsZt uE LrWc dtddddddddddddd }td d |it|fi|}|S) NrrrrTrFr"rrr)"vit_base_patch16_rope_reg1_gap_256rrs rVrrTsYJ  r rW[\fWqjpWq rE LrWc tddddddddddd d d dttd }tdd|it|fi|S)NrrrrrrXT)rrrxyr~repsrrrr-rrrrrrrrrrr5r)vit_pe_core_base_patch16_224rrrrrrrs rVr r isb!%9$/J$  k* kPTU_PjciPj kkrWc tdddddddddd d d d dttd }tdd|it|fi|S)NrrrrrrrXT)rrrrr~rr r r)vit_pe_core_large_patch14_336r rs rVrrsb!%9$/J$  l: lQUV`QkdjQk llrWc tddddddddd d d d d d ttd}tdd|it|fi|S)Nr2rUUUUUU@rrXFT rrr~rrr )rrrr-rrrrrrrrrrr5r) vit_pe_core_gigantic_patch14_448r rs rVrrsb!%9$/J$  oj oTXYcTngmTn oorWc td iddddddddd d d d d ddddddddddddddddddttd}td!d|it|fi|S)"Nrrrrrr-rrrrrXrTrrrrrrrrrFrr皙?r5rr rr)vit_pe_lang_large_patch14_448r rs rVrrs       "&#( 9$/!J&  l: lQUV`QkdjQk llrWc tddddddddd d dddd ttd }tdd|it|fi|S)Nrr/rrrXFTrrrrr rrrr-rrrrrrrrrrr5r) vit_pe_lang_gigantic_patch14_448r rs rVrrsb!%"'9$/J$  oj oTXYcTngmTn oorWc tddddddddd d dddd ttd }tdd|it|fi|S)NrrrrrrXFTrrrrr rr)#vit_pe_spatial_gigantic_patch14_448r rs rVr!r!sb!%"'9$/J$  r rW[\fWqjpWq rrrW)r)rTrj)r)]rr functoolsrtypingrrrrrr r r rGtorch.nnr@torch.nn.functional functionalre timm.datar r rr timm.layersrrrrrrrrrrrrrrrrr r!_builderr# _featuresr$ _manipulater% _registryr&r'__all__rkr*rrr(rrrrprrrr default_cfgsrrrrrrrrrrrrrrrrrrrrr rrrrr!rrWrVr/s 4 III ddSSSSS++#< 'G299GTdryydN^ryy^B"))JCS%,,&'C 99C C #u||  CR' \ell*+\yy\\ \  #u||  \~$S*cT#s(^(S#X(%g&)$+g&)$ 3(+Dg&/ "(< 3( 1Dg&$/ "(< 3( 1D%g&25d 3(73g&<6t 3(8=g&F5d 3(7Gg&T/ 31Ug&^0 32_g&h/ 31ig&r0 32sg&|/ 31}g&J0 3(PU2Kg&T1$ 3(PU3Ug&^0 3(PU2_g&l')mg&v(*wg&@')Ag&J(*Kg&T')Ug&b+D -cg&p*4 ,qg&~+D -g&L,T .Mg&Z,T 3 .[g&j.t 0kg&x3D 5yg&F/1Gg&P4T 4 /6Qg&Z5d 4 /7[g&d5d 47eg&l24 4 /4mg&z&w! ({g&H'! )Ig&V,3! ,)0! ),3! ,/6! /Ag&g Tddd t #     3   t#$3$ t #     3   t#$3$4cDs*T*T*  3   TPS(dQT(dQT(4c(lTlll,mdmmm,ppCpp,mdmmm.ppCpp,sDssssrW