kh dZddlZddlZddlmZmZmZmZmZm Z m Z m Z ddl Z ddl mZddlmZmZddlmZmZmZmZmZmZmZmZmZmZmZddlmZddl m!Z!dd l"m#Z#dd l$m%Z%m&Z&dd l'm(Z(m)Z)m*Z*dd l+m,Z,d gZ-ej\e/Z0e e1e e1e1ffZ2de jfde e1e1fde jffdZ4e#de jfde e1e1fde1de1de jff dZ5de1de1de jffdZ6GddejnZ8GddejnZ9GddejnZ:Gdd ejnZ;Gd!d ejnZe jfffd$Z?dbd%e>d&e@dedee>effd)ZBe(id*eBd+d,-d.eBd+d/-d0eBd+d1d2d3d45d6eBd+d7-d8eBd+d9d2d3d45d:eBd+d;-d<eBd+d=-d>eBd+d?-d@eBd+dAd2d3d45dBeBd+dC-dDeBd+dEdFGdHeBd+dIdFGdJeBd+dKdFGdLeBd+dMd2d3d4dFNdOeBd+dPdFGdQeBd+dRd2d3d4dFNdSeBd+dT-eBd+dU-eBd+dV-dWZCe)dbde.A;q>S^_`SacdeA !Q1a#..055b!QBA Hr9win_hwin_wctjttj|tj|}tj|d}|dddddf|dddddfz }|j dddj }|dddddfxx|dz z cc<|dddddfxx|dz z cc<|dddddfxxd|zdz zcc<|jdS)zGet pair-wise relative position index for each token inside the window. Args: win_h: Window height. win_w: Window width. Returns: Relative position index tensor. rNr)rr,)torchstackrarangeflattenr0r1sum)r<r=coordscoords_flattenrelative_coordss r7get_relative_position_indexrGPs[[ U 3U\\%5HI JF]]61-N$Q4Z0>!T1*3MMO%--aA6AACOAq!G )Aq!G )Aq!GE A -   r ""r9ceZdZUdZej j eed< dde de de e de ded e d e ffd Z dee e fd dfd Zd ej fdZddej de ej d ej fdZxZS)WindowAttentionzWindow based multi-head self attention (W-MSA) module with relative position bias. Supports both shifted and non-shifted windows. fused_attnNdim num_headshead_dimr%qkv_bias attn_drop proj_dropct |||_t||_|j\}} || z|_||_|xs||z}||z} |dz|_td|_ tjtjd|zdz d| zdz z||_|jdt!|| dtj"|| d z| |_tj&||_tj"| ||_tj&||_t/|jd tj0d |_y)a Args: dim: Number of input channels. num_heads: Number of attention heads. head_dim: Number of channels per head (dim // num_heads if not set) window_size: The height and width of the window. qkv_bias: If True, add a learnable bias to query, key, value. attn_drop: Dropout ratio of attention weight. proj_drop: Dropout ratio of output. gT) experimentalr)rrelative_position_indexF persistentr(biasg{Gz?)stdr,)rKN)super__init__rKrr% window_arearLscalerrJnn Parameterr?zerosrelative_position_bias_tableregister_bufferrGLinearqkvDropoutrOprojrPrSoftmaxsoftmax) selfrKrLrMr%rNrOrPr<r=attn_dim __class__s r7rZzWindowAttention.__init__ls>( $[1'' u 5="/si/i'% (d;-/LLa%iRSmXY\aXadeXeEfhq9r,s) 68STY[`8anst99S(Q,X>I.IIh, I.d77SAzzb) r9r&ct|}||jk(ry||_|j\}}||z|_tj5d|zdz d|zdz z|j f}t jt|j|j||_ |jdt||ddddy#1swYyxYw)zzUpdate window size & interpolate position embeddings Args: window_size (int): New window size Nr)rnew_window_sizenew_bias_shaperSFrT) rr%r[r?no_gradrLr]r^rr`rarG)rhr%r<r=rns r7set_window_sizezWindowAttention.set_window_sizes  , $** * &'' u 5= ]]_ y%i!mE A >NN02 )55$($4$4#11D -  !:>$'DqA KK1  % %b!R 0 IIaL NN1 r9)NTrwrwN)__name__ __module__ __qualname____doc__r?jitFinalbool__annotations__intr_int_or_tuple_2_tfloatrZr rpTensorrtr __classcell__rjs@r7rIrIes %% '+-.!!!+*+*+*sm +* + +*  +*+*+*Zy5c?yty*35<<3 ((Xell-C(u||(r9rIceZdZdZddddddddd d d ej ej f d ed ed ed e ededede de de de de de de de de ffd Z d$de ejde ejfdZ d$deeeeeffde eeeeeffdeeeefeeefffdZ d$d eeefdeeefde e fd!Zd"Zdejdejfd#ZxZS)%SwinTransformerBlockzkSwin Transformer Block. A transformer block with window-based self-attention and shifted windows. r*NrrF@TrwrKinput_resolutionrLrMr% shift_sizealways_partition dynamic_mask mlp_ratiorNrPrO drop_path act_layer norm_layerc t|||_||_t ||_||_||_|j||\|_ |_ |jd|jdz|_ | |_ |||_ t||||j| | | |_| dkDr t!| nt#j$|_|||_t+|t-|| z|| |_| dkDr t!| nt#j$|_|j3d|jrdn|j5d y) a Args: dim: Number of input channels. input_resolution: Input resolution. window_size: Window size. num_heads: Number of attention heads. head_dim: Enforce the number of channels per head shift_size: Shift size for SW-MSA. always_partition: Always partition into full windows and shift mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. proj_drop: Dropout rate. attn_drop: Attention dropout rate. drop_path: Stochastic depth rate. act_layer: Activation layer. norm_layer: Normalization layer. rr)rLrMr%rNrOrPrw) in_featureshidden_featuresrdroprxNFrT)rYrZrKrrtarget_shift_sizerr_calc_window_shiftr%rr[rnorm1rIrrr]Identity drop_path1norm2rrmlp drop_path2ra get_attn_mask)rhrKrrLrMr%rrrrrNrPrOrrrrjs r7rZzSwinTransformerBlock.__init__sIF  0!*:!6 0(,0,C,CKQ[,\)$/++A.1A1A!1DD"_ # ((  2;R(9-R[[]_ i0   2;R(9-R[[]  %%D4+=+=+?  r9r$r&c dt|jr|7|jd|jd}}|j}|j}n|j \}}d}d}t j||jdz |jdz}t j||jdz |jdz}tjd||df||}d}d|jd f|jd |jd f|jd dffD]l}d|jd f|jd |jd f|jd dffD]$} ||dd|d|d| d| dddf<|dz }&nt||j} | jd|j} | jd| jdz } | j| dk7t!dj| dk(t!d} | Sd} | S)Nrr)r)dtypedevicer,gYrw)anyrr.rrrmathceilr%r?r_r8r/r[rr masked_fillr) rhr$r3r4rrimg_maskcnthw mask_windowsrxs r7rz"SwinTransformerBlock.get_attn_mask-sZ t }wwqz1771:1,,1 !d..q112T5E5Ea5HHA !d..q112T5E5Ea5HHA{{Aq!Q+>?ooa(($/  T--a001**1--0B/BC//!,,d3A '>>$BRBRSTBUU!!!$q4+;+;A+>'>>$BRBRSTBUUHH''++I1a57QR   2r1%Y0@0@A NN2t'7'7;  4 /**95IIyyy; $((T-=-=a-@$BRBRSTBUWXY "<1A1A2rJ a!RaRl+668   9T__6JAAr9c>|j\}}}}||j|j|j|z}|j |d|}||j |j |j|z}|j ||||}|S)zForward pass. Args: x: Input features with shape (B, H, W, C). Returns: Output features with shape (B, H, W, C). r,)r.rrrr{rrr)rhr$r2r3r4r5s r7rzSwinTransformerBlock.forwardsWW 1a  4::a= 9: : IIaQ  A 78 8 IIaAq !r9r)rrrrr]GELU LayerNormrrrrrrrZr?rrr r rrrrrrs@r7rrs&*-.%*!&!!!!!"$''#%<z0SwinTransformerStage.__init__..s&H!qAv&HFr))rKrrr)rKrrLrMr%rrrrrNrPrOrrN)rYrZrKrroutput_resolutionrgrad_checkpointingrrrr]r Sequentialranger isinstancelistblocks)rhrKrrrrrLrMr%rrrrNrPrOrrrrrrjs r7rZzSwinTransformerStage.__init__s*H  0LV&H7G&H!H\l "' , K8qAF89  *%DO '> !> kkmDOmm"5\#&#"! !!%!7!7#!'!"Q!1*!1)#!##*4Y*E)A,9% &#$ 9&#s  DA Drc||_t|jtjr||_nt d|D|_|jD] }|j|j ||"y)a Updates the resolution, window size and so the pair-wise relative positions. Args: feat_size: New input (feature) resolution window_size: New window size always_partition: Always partition / shift the window c3&K|] }|dz ywrrrs r7rz6SwinTransformerStage.set_input_size..Es*Ea16*Errr%rN) rrrr]rrrrr)rhrr%rblocks r7rz#SwinTransformerStage.set_input_size4sn!* door{{ 3%.D "%**E9*E%ED "[[ E  00'!1 !  r9r$r&c|j|}|jr6tjj st |j |}|S|j |}|S)zsForward pass. Args: x: Input features. Returns: Output features. )rrr?r is_scriptingrrrhr$s r7rzSwinTransformerStage.forwardMsU OOA   " "599+A+A+Ct{{A.A AAr9r)rrrrr]rrr rrrrr rrrZrr?rrrrs@r7rrsN $&*-.%*!&!!!!35#%<<#J$J$J$$CHo J$  J$  J$J$smJ$+J$#J$J$J$J$J$J$ T%[%/0!J$"!#J$`04 S#X'tn 2%,,r9rc+`eZdZdZdddddddd d d d d dd ddddeej dfdedededede dede edfde edfde edede de de d e d!e d"e d#e d$e d%ed&ee efd'e f*fd( Zej$j&dCd)e d*d fd+Zej$j&d*ee fd,Z dDde e eefde e eefde e eefd-ede e d*d f d.Zej$j&dEd/e d*ee effd0Zej$j&dFd1e d*d fd2Zej$j&d*ej8fd3ZdGdede e d*d fd4Z dHd5ej>d6e eee efd7e d8e d9e d:e d*ee ej>e ej>e ej>fffd;Z! dId6eee efdZ"d5ej>d*ej>fd?Z#dEd5ej>d@e d*ej>fdAZ$d5ej>d*ej>fdBZ%xZ&S)Jr#zSwin Transformer. A PyTorch impl of : `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows` - https://arxiv.org/pdf/2103.14030 r*r(avg`r)r)r)r(r NrFTrrwg?img_size patch_sizein_chans num_classes global_pool embed_dimdepths.rLrMr%rstrict_img_sizerrN drop_rateproj_drop_rateattn_drop_ratedrop_path_rate embed_layerr weight_initc t||dvsJ||_||_d|_t ||_||_t|d|j dz zzx|_ |_ g|_ t|ttfs1t|j Dcgc]}t|d|zz}}|||||d|| d|_|j j"}t%|j | } t| ttfst%|j | } nt | dk(r| f|j z} t | |j k(sJt%|j | } t'j(d|t+|j-|Dcgc]}|j/}}g}|d}d}t|j D]}||}|t1did|d|d |d|z|d|zfd ||d |dkDd ||d | |d| |d| d| d| |d|d|d|d||d|gz }|}|dkDr|dz}|xjt3|||zd|gz c_ t5j6||_||j|_t=|j||||j|_|dk7r|jA|yycc}wcc}w)a~ Args: img_size: Input image size. patch_size: Patch size. in_chans: Number of input image channels. num_classes: Number of classes for classification head. embed_dim: Patch embedding dimension. depths: Depth of each Swin Transformer layer. num_heads: Number of attention heads in different layers. head_dim: Dimension of self-attention heads. window_size: Window size. mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. drop_rate: Dropout rate. attn_drop_rate (float): Attention dropout rate. drop_path_rate (float): Stochastic depth rate. embed_layer: Patch embedding layer. norm_layer (nn.Module): Normalization layer. )rrNHWCr)rr)r r r rrr output_fmtrKrrrrrLrMr%rrrrNrPrOrrlayers.)num_chsrmodule) pool_typer input_fmtskipNr)!rYrZr r rlen num_layersrr num_featureshead_hidden_size feature_inforrrr patch_embed grid_sizerr?linspacerCsplittolistrdictr]rlayersrrhead init_weights) rhr r r r r rrrLrMr%rrrrNrrrrrrrkwargsr patch_gridr$dprr+in_dimr\rrjs r7rZzSwinTransformer.__init__fs}X k)))&& f+"47 A$//\]J]D^8^4__D1)eT]3:?:PQQYa/0QIQ'!l!+ %%// .9T__-h7+e}54)DOO4[AK   "&.4??:K;4??222.Idoo.y9 #(>>!^S[#Q#W#WX^#_`aqxxz``1t' mAlG +qMU*qMU*"Qiq5$A,"!(N"2"10$A," )!")#$a&%&&' F*F1u    $w*uBT_fghfi]j"k!l l 5 m6mmV, t001 "    !oo    &   k * !R,as K=8Lmoder&c|dvsJd|vr tj|j nd}tt |||y)zInitialize model weights. Args: mode: Weight initialization mode ('jax', 'jax_nlhb', 'moco', or ''). )jaxjax_nlhbmocornlhbrw) head_biasN)rlogr rr")rhr2r8s r7r-zSwinTransformer.init_weightssD666639T>TXXd..//r (CTJr9cvt}|jD]\}}d|vs |j||S)z,Parameters that should not use weight decay.r`)setnamed_parametersadd)rhnwdnrs r7no_weight_decayzSwinTransformer.no_weight_decaysAe))+ DAq-2   r9 window_ratiocZ||3|jj|||jj}|tDcgc]}||z c}}t |j D]9\}} dt |dz dz} | jd| z|d| zf||;ycc}w)aUpdate the image resolution and window size. Args: img_size: New input resolution, if None current resolution is used. patch_size: New patch size, if None use current patch size. window_size: New window size, if None based on new_img_size // window_div. window_ratio: Divisor for calculating window size from grid size. always_partition: Always partition into windows and shift (even if window size < feat size). N)r r r)rrr)r%rr&r enumerater+max) rhr r r%rArr/pgindexstage stage_scales r7rzSwinTransformer.set_input_sizes"  :#9    + +X* + U))33J  j I|!3 IJK%dkk2 LE5s519a00K  %a=K7A+9UV'!1 !  !Js B(coarsec2td|rdSgdS)z"Group parameters for optimization.z ^patch_embedz^layers\.(\d+)))z^layers\.(\d+).downsample)r)z^layers\.(\d+)\.\w+\.(\d+)N)z^norm)i)stemr)r*)rhrIs r7 group_matcherzSwinTransformer.group_matchers) (.$  5  r9enablec4|jD] }||_ y)z)Enable or disable gradient checkpointing.N)r+r)rhrMls r7set_grad_checkpointingz&SwinTransformer.set_grad_checkpointing s *A#)A  *r9c.|jjS)zGet the classifier head.)r,fc)rhs r7get_classifierzSwinTransformer.get_classifier&syy||r9cL||_|jj||y)zReset the classifier head. Args: num_classes: Number of classes for new classifier. global_pool: Global pooling type. )rN)r r,reset)rhr r s r7reset_classifierz SwinTransformer.reset_classifier+s '  {;r9r$indicesr stop_earlyrintermediates_onlyc>|dvsJdg}tt|j|\}} |j|}t|j} tj j s|s |j} n|jd| dz} t| D]c\} } | |}| |vs|r| | dz k(r|j|}n|}|jddddj}|j|e|r|S|j|}||fS)aKForward features that returns intermediates. Args: x: Input image tensor. indices: Take last n blocks if int, all if None, select matching indices if sequence. norm: Apply norm layer to compatible intermediates. stop_early: Stop iterating over blocks when last desired intermediate hit. output_fmt: Shape of intermediate feature outputs. intermediates_only: Only return intermediate features. Returns: List of intermediate features or tuple of (final features, intermediates). )NCHWzOutput shape must be NCHW.Nrrr(r)) rr r+r%r?rrrCrr0r1append)rhr$rWrrXrrY intermediates take_indices max_index num_stagesstagesrrGx_inters r7forward_intermediatesz%SwinTransformer.forward_intermediates5s,Y&D(DD& "6s4;;7G"Q i   Q % 99 ! ! #:[[F[[)a-0F!&) .HAuaAL Aa/"iilGG!//!Q15@@B$$W- .  IIaL-r9 prune_norm prune_headctt|j|\}}|jd|dz|_|rtj|_|r|j dd|S)aEPrune layers not required for specified intermediates. Args: indices: Indices of intermediate layers to keep. prune_norm: Whether to prune normalization layer. prune_head: Whether to prune the classifier head. Returns: List of indices that were kept. Nrrr)rr r+r]rrrV)rhrWrdrer^r_s r7prune_intermediate_layersz)SwinTransformer.prune_intermediate_layershs] #7s4;;7G"Q ikk.9q=1  DI   ! !!R (r9cl|j|}|j|}|j|}|S)z/Forward pass through feature extraction layers.)r%r+rrs r7forward_featuresz SwinTransformer.forward_featuress1   Q  KKN IIaLr9 pre_logitscN|r|j|dS|j|S)zForward pass through classifier head. Args: x: Feature tensor. pre_logits: Return features before final classifier. Returns: Output tensor. T)rj)r,)rhr$rjs r7 forward_headzSwinTransformer.forward_heads&1;tyyty,L ! Lr9cJ|j|}|j|}|S)zoForward pass. Args: x: Input tensor. Returns: Output logits. )rirlrs r7rzSwinTransformer.forwards)  ! !! $   a r9r)NNNNF)Tr)NFFr[F)rFT)'rrrrrr]rrrstrr rrrrr rZr?rignorer-rr@rrrrLrPModulerSrVrrrcrgrirlrrrs@r7r#r#_s+.#$&2)7&*-.%*$(!!!$&$&$'$./1||!-x+'x+x+ x+  x+  x+x+#s(Ox+S#Xx+smx++x+#x+"x+x+x+ !x+""#x+$"%x+&"'x+(")x+*c8m,+x+,-x+t YYKKdKK YYS374859 !/3 uS#X/!sCx1"%S/2    'tn  @ YY  D  T#s(^     YY*T*T**  YY *  c 0%,,5<< Mell M M M  %,, r9 state_dictmodelc2d}d|vrd}ddl}i}|jd|}|jd|}|jD]K\}}tdDcgc]}||vc}r#d |vre|jj j j\}}} } |jd | k7s|jd | k7rt|| | fd dd }|jdr|j|dd} |j| jjk7s| jd| jdk7r,t|| j| jj}|r&|jdd|}|jdd}|||<N|Scc}w)zConvert patch embedding weight from manual patchify + linear proj to conv. Args: state_dict: State dictionary from checkpoint. model: Model instance. Returns: Filtered state dictionary. Tzhead.fc.weightFrNrurt)rSrxzpatch_embed.proj.weightrzr,bicubic) interpolation antialiasverboser`irrlzlayers.(\d+).downsamplecDdt|jddzdS)Nrrz .downsample)rgroup)r$s r7z&checkpoint_filter_fn..s$ws177ST:YZGZF[[f=gr9zhead.zhead.fc.)regetitemsrr%reweightr.rendswith get_submoduler`r%rsubreplace) rtru old_weightsr~out_dictrrr?rr3r4ms r7checkpoint_filter_fnrsK:%  H4J j9J  "1  HI1QI J  $ )**//66<>>!--PQBRVWVcVcdeVfBf-$%MM#$#A#A#G#G 13gijkA ':.A 9: O9Js F variant pretrainedc tdt|jddD}|jd|}t t ||ft td|d|}|S)zCreate a Swin Transformer model. Args: variant: Model variant name. pretrained: Load pretrained weights. **kwargs: Additional model arguments. Returns: SwinTransformer model instance. c3&K|] \}}| ywrr)rrrs r7rz+_create_swin_transformer..s\da\rr)rrr(r out_indicesT)flatten_sequentialr)pretrained_filter_fn feature_cfg)rrCrpoprr#rr*)rrr.default_out_indicesrrus r7_create_swin_transformerrsj \i 8\8Z.[\\**],?@K * 1DkJ    E Lr9urlc 4|ddddddttddd d |S) z9Create default configuration for Swin Transformer models.r)r(rr)rrg?rwTzpatch_embed.projzhead.fcmit) rr  input_size pool_sizecrop_pctrxfixed_input_sizemeanrX first_conv classifierlicenser )rr.s r7_cfgrs7=v%.B(    #  r9z.swin_small_patch4_window7_224.ms_in22k_ft_in1kztimm/zvhttps://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_small_patch4_window7_224_22kto1k_finetune.pth) hf_hub_idrz-swin_base_patch4_window7_224.ms_in22k_ft_in1kzlhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22kto1k.pthz.swin_base_patch4_window12_384.ms_in22k_ft_in1kzmhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384_22kto1k.pth)r(r)rrg?)rrrrrz.swin_large_patch4_window7_224.ms_in22k_ft_in1kzmhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22kto1k.pthz/swin_large_patch4_window12_384.ms_in22k_ft_in1kznhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window12_384_22kto1k.pthz$swin_tiny_patch4_window7_224.ms_in1kzdhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_tiny_patch4_window7_224.pthz%swin_small_patch4_window7_224.ms_in1kzehttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_small_patch4_window7_224.pthz$swin_base_patch4_window7_224.ms_in1kzdhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224.pthz%swin_base_patch4_window12_384.ms_in1kzehttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384.pthz-swin_tiny_patch4_window7_224.ms_in22k_ft_in1kzuhttps://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_tiny_patch4_window7_224_22kto1k_finetune.pthz%swin_tiny_patch4_window7_224.ms_in22kzhhttps://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_tiny_patch4_window7_224_22k.pthiQU)rrr z&swin_small_patch4_window7_224.ms_in22kzihttps://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_small_patch4_window7_224_22k.pthz%swin_base_patch4_window7_224.ms_in22kzhhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22k.pthz&swin_base_patch4_window12_384.ms_in22kzihttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384_22k.pth)rrrrrr z&swin_large_patch4_window7_224.ms_in22kzihttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22k.pthz'swin_large_patch4_window12_384.ms_in22kzjhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window12_384_22k.pthzswin_s3_tiny_224.ms_in1kzbhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/s3_t-1d53f6a8.pthzbhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/s3_s-3bb4c69d.pthzbhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/s3_b-a1e95db4.pth)zswin_s3_small_224.ms_in1kzswin_s3_base_224.ms_in1kc Rtddddd}t dd|it|fi|S) z+ Swin-T @ 224x224, trained ImageNet-1k r*rrrrr r%rrrLr)swin_tiny_patch4_window7_224r*rrr. model_argss r7rr@sFR `noJ #& ]3= ]AEjA[TZA[ ]]r9c Rtddddd}t dd|it|fi|S) z Swin-S @ 224x224 r*rrr)r)r)rrr)swin_small_patch4_window7_224rrs r7rrIsFR aopJ #' ^4> ^BFzB\U[B\ ^^r9c Rtddddd}t dd|it|fi|S) z Swin-B @ 224x224 r*rrr*ro rr)swin_base_patch4_window7_224rrs r7rrRsFSbpqJ #& ]3= ]AEjA[TZA[ ]]r9c Rtddddd}t dd|it|fi|S) z Swin-B @ 384x384 r*rrrrrr)swin_base_patch4_window12_384rrs r7rr[sFc-cqrJ #' ^4> ^BFzB\U[B\ ^^r9c Rtddddd}t dd|it|fi|S) z Swin-L @ 224x224 r*rrrrr0rr)swin_large_patch4_window7_224rrs r7rrdsFSbqrJ #' ^4> ^BFzB\U[B\ ^^r9c Rtddddd}t dd|it|fi|S) z Swin-L @ 384x384 r*rrrrrr)swin_large_patch4_window12_384rrs r7rrmsFc-crsJ #( _5? _CG C]V\C] __r9c Ptddddd}tdd|it|fi|S) z; Swin-S3-T @ 224x224, https://arxiv.org/abs/2111.14725 r*rrrrrrrr)swin_s3_tiny_224rrs r7rrvs;-2l^lnJ # l: lQUV`QkdjQk llr9c Ptddddd}tdd|it|fi|S) z; Swin-S3-S @ 224x224, https://arxiv.org/abs/2111.14725 r*)rrrrrrrrr)swin_s3_small_224rrs r7rrs;/R aoqJ # mJ mRVWaRlekRl mmr9c Ptddddd}tdd|it|fi|S) z; Swin-S3-B @ 224x224, https://arxiv.org/abs/2111.14725 r*rr)r)r)r)rrr)swin_s3_base_224rrs r7rrs;-2m_moJ # l: lQUV`QkdjQk llr9)"swin_base_patch4_window7_224_in22k#swin_base_patch4_window12_384_in22k#swin_large_patch4_window7_224_in22k$swin_large_patch4_window12_384_in22krprn)Mrloggingrtypingrrrrrrr r r?torch.nnr] timm.datar r timm.layersrrrrrrrrrrr_builderr _featuresr _features_fxr _manipulaterr _registryrr r!vision_transformerr"__all__ getLoggerr_loggerrrrr8r;rGrsrIrrrr#r*rqrrrr default_cfgsrrrrrrrrrrr9r7rs2 " III ALLLL*+34YY4   '  H %#uS#X./ <<38_ \\& ELL uS#X 3 SV [`[g[g  $#s#3#5<<#*wbiiwtS299Sl*299*Zz299zzbiiD .T.")).S%,,EV@W.bctRa. c T#s(^ %H&4d E7HH&4T z6} H&5d { Hs7DH&5d {7~H&6t | Hs8DH&&+D r-u'H&,,T s.v-H&2+D r-u3H&8,T s Hs.D9H&D4T D6FEH&L,T v.MH&T-d w/UH&\,T v.]H&d-d w HsPU/WeH&l-d w/mH&t.t x HsPU0WuH&~ p!rH&D"& p"r!% p!rKH&H V]]]^^^]]]^^^^^^_/__mOmmn_nnmOmmH*Q+S+S,U 'r9