kh dZddlZddlmZddlmZmZmZddlm Z ddl m Z m Z m Z mZmZmZmZmZddlZddlmZddlmZdd lmZmZdd lmZmZmZmZmZm Z dd lm!Z!m"Z"m#Z#m$Z$m%Z%m&Z&dd lm'Z'm(Z(m)Z)m*Z*m+Z+dd lm,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2ddl3m4Z4ddl5m6Z6ddl7m8Z8m9Z9ddl:m;Z;meGddZ?eGddZ@GddejZBGddejZCGdd ejZDGd!d"ejZEGd#d$ejZFdd&ejd'eGd(eGd)dfd*ZHGd+d,ejZIdd&ejd'eGd(eGd)dfd-ZJd.eeKd/eKd)eKfd0ZLGd1d2ejZMGd3d4ejZNd5ejd6eeKd)ejfd7ZPe6d8ejd6eeKd9eeKd)ejfd:ZQd5ejd;eeKd)ejfd<ZRe6d8ejd;eeKd9eeKd)ejfd=ZSd>e>d6eeKeKfd)ee fd?ZTGd@dAejZUGdBdCejZVd5ejd6eeKd)ejfdDZWe6d8ejd6eeKd9eeKd)ejfdEZXd5ejd;eeKd)ejfdFZYe6d8ejd;eeKd9eeKd)ejfdGZZGdHdIejZ[GdJdKejZ\GdLdMejZ]GdNdOejZ^GdPdQejZ_d>e>d9eeKeKfd)e>fdRZ`d>e@dSe d)e@fdTZaGdUdVejZb dd\eGd]eGd^ecd_ecd`eGdaeGdbecdceGddeGdeeeddfeGdgeKd)e eGe ffdhZe dd\eGd]eGd^ecdjeddaeGdceGddeGd6eeeKeKfdkeKdeeeddfeGdgeKd)e eGe ffdlZf dd\eGd]eGdaeGdoeGdceGddeGd6eeeKeKfdpecdeeedeededffdfeGdgeKd)e eGe ffdqZgd)e eGe ffdrZheididse@ddtdudvdwefdYdxydze@ddtd{dvdwefdWdYdx|d}e@dd~ddvdweedYdXde@dd~ddvdweedidYdXde@dddddweedidde@dddddweediddmde@dd~ddvdweedidYdXdde@ddtd{dvdwefdYdxdndde@dd~ddvdweedidnde@dd~ddvdweeddYdXdndde@dd~ddvdweedidnd[de@dddddweediddmdnde@dddddweediddmdnde@ddtd{dvddeede@ddtd{dvddegdZdde@d~duddde@d~dddde@ddddde@ddddde@ddddde@ddddde@ddddddefde@ddtdddvdefde@ddtdddvdefde@ddtdddvdefde@ddddddefdnde@ddtdddvdefdnde@ddtdddvdefdnde@dd~dddvdefdndmde@dd~dddvddƜefdnde@ddtdddvddȜegde@ddtdddvdegde@dd~ddddegde@dd~dddddȜegdYdZͫde@ddddddegdYЫde@dddddddƜegdYЫde@ddtddddYd[dלehde@dd~ddddYddלehde@dd~ddddYddלehde@ddddddYddלehde@ddddddYddלehZjde eGejfdejd)e eGejffdބZkddeGdeeGdecdSe d)ebf dZlddeGdSe d)e eGe ffdZme;idemd%demddddemdddemdddemddemddemddemddddddemddeeddemddddemd%demdddemdddemd%demd%demdd dd emdd  id emdd  demdd  demdd  demd%demd%demd%demd%demd%demd%demd%dddemdddddemdddemd%ddd emd%ddd!emdd"ddd#emdd$ddd%emdd&ddid'emdd(dd)emd%ddd*emdd+emdddddd,emdd  d-emdd.ddd/emd%ddd0emdd1ddd2emddd3d4emdd5emdddddd6emd%d7emdd  d8emdee9d:emdddddd;emdd<d=ddd>emdee9id?emdddddd@emdd<d=dddAemdee9dBemddddddCemdd<d=dddDemdee9dEemddddddFemdd<d=dddGemddH dIemddddddJemdd<d=dddKemddH dLemddddddMemddnorm_epsc|jt|j|_|j9t|j|_|j|j|_yyyN)rJr!rIselfs O/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/models/maxxvit.py __post_init__z#MaxxVitTransformerCfg.__post_init__Vs\ >> %&t~~6DN    '()9)9:D ~~%!%!1!1& ()!__name__ __module__ __qualname____doc__r8int__annotations__r9boolr;floatr<r=r>r@rArCstrrErGrHrIr rrJrKrLrMrOrQrSrUr[r\rZr4r4=s7HcJL%L$M4ItIuIuIsL#KOS-1K%S/*1+/Ixc3h(/M4M4#'K%'Is#J#$M3$He2r\r4c<eZdZUdZdZeed<dZeed<dZ e ed<dZ e ed <d Z e ed <d Ze ed <dZe ed<dZeed<dZeed<dZeed<dZeed<d Ze ed<dZeed<dZeed<dZeed<dZeeed<dZeed <dZeed!<dZeed"<d#Zeeed$<d%Zy#)&r3z-Configuration for MaxxVit convolution blocks.mbconv block_typer:r;T expand_output kernel_sizer* group_sizeF pre_norm_act output_biasdw stride_moderBrCdownsample_pool_typepadding attn_earlyse attn_layersiluattn_act_layer? attn_ratiorTrMrNrOrQrSNrUc|jdvsJ|jdk(}|js |rdnd|_|js |sd|_|j |rdnd|_|jxs |j |_y)N)rhconvnextrh batchnorm2drPrRh㈵>rT)rirQrSrUrrrC)rY use_mbconvs rZr[zMaxxVitConvCfg.__post_init__wst"8888__0 /9m}DO!!*!,D  == $.DDDM$($=$=$O!r\)r]r^r_r`rirerbr;rdrjrcrlrarmrnrorqrCrrrtrurwryr{rMr rOrQrSrUr[rfr\rZr3r3_s7JL%M4KJL$KKIs &#&GSJJ NC J#'K%'IsJM3 $Hhuo$ Pr\r3ceZdZUdZdZeedfed<dZeedfed<dZ ee e ee dffdfed<d Z e eeeeffed <d Z eed <ee Zeed<ee Zeed<dZeeed<dZe ed<y)r2z!Configuration for MaxxVit models.`. embed_dimrkrdepths)CrTrri@ stem_widthF stem_bias)default_factoryconv_cfgtransformer_cfgNhead_hidden_sizevit_eff weight_init)r]r^r_r`rrrarbrrirrerrrcrr3rr4rrr rrfr\rZr2r2s+!4IuS#X4*FE#s(O*:NJeCsCx01367N.0Jc5c?*+0It$^DHnD-2CX-YO*Y&*hsm* K r\r2ceZdZUdZeeed< ddedeededededed ee d e d e ffd Z dd e jdee jde jfdZxZS) Attention2dz)Multi-head attention for 2D NCHW tensors. fused_attndimdim_outr8rDr<r9 rel_pos_clsr@rAc t ||xs|}|r|n|} | |z|_||_||_|dz|_t |_tj|| dzd||_ |r||jnd|_ tj||_ tj| |d||_tj| |_y) Args: dim: Input dimension. dim_out: Output dimension (defaults to input dimension). dim_head: Dimension per attention head. bias: Whether to use bias in qkv and projection. expand_first: Whether to expand channels before or after qkv. head_first: Whether heads are first in tensor layout. rel_pos_cls: Relative position class to use. attn_drop: Attention dropout rate. proj_drop: Projection dropout rate. rkr*rD num_headsN)super__init__rr8r9scaler(rrConv2dqkvrel_posDropoutr@projrA rYrrr8rDr<r9rr@rAdim_attn __class__s rZrzAttention2d.__init__s0 .S*7!X-  $% (*99S(Q,=@K{T^^ I.r\xshared_rel_posreturnc|j\}}}}|jrP|j|j||j|j dzdj dd\}}} nK|j|j|d|j|j djd\}}} |jrd} |j|jj} n||} tjjj|j!ddj#|j!ddj#| j!ddj#| |j$r|j&j(ndj!ddj|d||}n||j*z}|j!dd|z} |j|j| } n|| |z} | j-d} |j'| } | | j!ddzj|d||}|j/|}|j1|}|S) Nrkrrr*r? attn_mask dropout_p)shaper9rviewrr8chunkreshapeunbindrrget_biastorchr functionalscaled_dot_product_attention transpose contiguoustrainingr@prsoftmaxrrA) rYrrBrHWqkvr>attns rZforwardzAttention2d.forwardsWW 1a ??hhqk&&q$..$--!:KRPVVWX^_V`GAq!hhqk))!Q rRYYZ[\GAq! ??I||' LL113 +* ##@@ B#..0 B#..0 B#..0#.2mm$..** A iB2q! 4 DJJA;;r2&*D||'||D)+n,<>$'DT^^B++11!RA>A IIaL NN1 r\Nr7TTTNr?r?rWr]r^r_r`rrcrbrar r rdrrTensorr __classcell__rs@rZrrs3d  &*!%#.2!!%/%/c]%/ %/  %/  %/%/"(+%/%/%/N##x 7M#Y^YeYe#r\rceZdZUdZeeed< ddedeededededed ee d e d e ffd Z dd e jdee jde jfdZxZS) AttentionClz/Channels-last multi-head attention (B, ..., C).rrrr8rDr<r9rr@rAc t ||xs|}|r||kDr|n|} | |zdk(sJd| |z|_||_||_|dz|_t |_tj|| dz||_ |r||jnd|_ tj||_ tj| |||_tj| |_y)rrz(attn dim should be divisible by head_dimrrkrrN)rrrr8r9rr(rrLinearrrrr@rrArs rZrzAttentionCl.__init__s0 .S*w}7#("a'S)SS'!X-  $% (*99S(Q,T:@K{T^^rs rZrzAttentionCl.forwards GGAJ ??hhqk&&q"dnndmma>OPZZ[\^_`ffghnofpGAq!hhqk))!RDNNDMMR\\]^`abiijklGAq! ??I||' LL113 +* ##@@1a#.2mm$..**AA DJJAq{{2r**D||'||D|H+n,<>$'DqA KK1  % %me&; < IIaL NN1 r\rrWrrs@rZrrs9d  &*!%#.2!!&/&/c]&/ &/  &/  &/&/"(+&/&/&/P##x 7M#Y^YeYe#r\rcjeZdZdZd dededeffd ZdejdejfdZ xZ S) LayerScalezPer-channel scaling layer.rrMinplacect|||_tj|t j |z|_yz Args: dim: Number of channels. init_values: Initial scaling value. inplace: Whether to perform inplace operations. Nrrrr ParameterronesgammarYrrMrrs rZrzLayerScale.__init__84  \\+ 3"?@ r\rrc^|j}|jr|j|S||zSrW)rrmul_rYrrs rZrzLayerScale.forwardCs(  $ qvve};!e);r\rF r]r^r_r`rardrcrrrrrrs@rZrr5s?$ AC Ae AT A<<%,,))C!$?DK++-DKr\rrcJ|j|}|j|}|SrW)rrrYrs rZrzDownsample2d.forwards! IIaL KKNr\)rBrsT) r]r^r_r`rarercrrrrrrs@rZrr[sd$ ((( (  (  (B%,,r\rrsmodulenameschemerct|tjtjfr|dk(rbtjj |j d|j*tjj|jyy|dk(rNt|j d|j*tjj|jyy|dk(r`tjj|j |j*tjj|jyytjj|j |jZd|vr,tjj |jdytjj|jyyy) z&Initialize transformer module weights.normal{Gz?stdN trunc_normal xavier_normalmlprT) isinstancerrrinitnormal_weightrDzeros_r xavier_normal_xavier_uniform_)rrrs rZ_init_transformerr s5&299bii01 X  GGOOFMMsO 3{{&v{{+' ~ % V]] 4{{&v{{+'  & GG " "6== 1{{&v{{+' GG # #FMM 2{{&D=GGOOFKKTO:GGNN6;;/ '!2r\c eZdZdZddedfdedededeed ed ef fd Z dd e d dfdZ dde jdee jd e jfdZxZS)TransformerBlock2daYTransformer block with 2D downsampling. '2D' NCHW tensor layout Some gains can be seen on GPU using a 1D / CL block, BUT w/ the need to switch back/forth to NCHW for spatial pooling, the benefit is minimal so ended up using just this variant for CoAt configs. This impl was faster on TPU w/ PT XLA than the 1D experiment. r*Nr?rrrrcfg drop_pathc t |tt|j|j }t |j}|dk(rnt|||j|j|_ tjtd||fdt|||jfg|_n-||k(sJtj |_ |||_t#|||j$|j&|j(||j*|j,|_|j0rt3||j0ntj |_|d kDr t7|ntj |_|||_t=|t?||j@z||j, |_!|j0rt3||j0ntj |_"|d kDrt7||_#y tj |_#y ) a Args: dim: Input dimension. dim_out: Output dimension. stride: Stride for downsampling. rel_pos_cls: Relative position class. cfg: Transformer block configuration. drop_path: Drop path rate. epsr)rCrDnormdownrC)r8r<rDrr@rArMr? in_featureshidden_featuresrOdropN)$rrrrrQrUrrOrrCr=shortcutr Sequentialrnorm1rrr8r<r>r@rArrMrls1r drop_path1norm2rrar;rls2 drop_path2) rYrrrrrrrQrOrs rZrzTransformerBlock2d.__init__s$ ^CNN;N !#--0 Q;(gUXUfUfgDM{C)c3#--HI4(DJ '> !>KKMDM#CDJ  \\))#mmmm   JM<S__E^`^i^i^k1:R(9-R[[]( #*:*: :;   JM<S__E^`^i^i^k1:R(9-R[[]r\rrc:ttt||yNr)r.rr rYrs rZ init_weightszTransformerBlock2d.init_weightssG-f=tDr\rrc ,|j||j|j|j|j ||z}||j |j |j|j|z}|S)Nr) rr!r rrr$r#rr")rYrrs rZrzTransformerBlock2d.forwardso MM! ttxx $**Q-`n 8o/pq q $**Q-)@ AB Br\rsrW)r]r^r_r`r4rar r rdrrer)rrrrrs@rZrrs.2)>)@!5S5S5S 5S "(+ 5S ' 5S5SnE3EEx 7MY^YeYer\rct|tjr|dk(rbtjj |j d|j *tjj|j yy|dk(rNt|j d|j *tjj|j yy|dk(r`tjj|j |j *tjj|j yy|jd|jdz|jz}||jz}tjj |j dtjd |z |j *tjj|j yyy) z&Initialize convolution module weights.rrrNrrrr*g@)rrrrrr rDr r r rl out_channelsgroupsmathsqrt)rrrfan_outs rZ _init_convr2sU&"))$ X  GGOOFMMsO 3{{&v{{+' ~ % V]] 4{{&v{{+'  & GG " "6== 1{{&v{{+'((+f.@.@.CCfFYFYYG  %G GGOOFMM1diig .F G{{&v{{+'%%r\rmchannelsc&|sy||zdk(sJ||zS)z3Calculate number of groups for grouped convolution.r*rrf)rmr3s rZ num_groupsr5s( *$))):%%r\c eZdZdZddedfdedededeeefd ed ef fd Zdd e d dfdZ de jd e jfdZ xZS) MbConvBlockzGPre-Norm Conv Block - 1x1 - kxk - 1x1, w/ inverted bottleneck (expand).r*r*r*r?in_chsout_chsrdilationrrc tt| tt |j |j |j}t|jr|n||jz}t|j|} |dk(r4t|||j|j|j |_nt%j&|_|j(dvsJd\} } } |j(dk(r||d} } n|j(dk(r||d} } n||d } } |||j* |_| dkDr)t|||j.|j |_nt%j&|_t3||d| |_|||_t3|||j8| | | |j |_i}t=|j>t@rV|j>dk(s|j>dk(r8|jB|d<tE|jF|jr|n|z|d<|jHr1tK|j>|fi||_&|||_'d|_(n0d|_&|||_'tK|j>|fi||_(t3||d|j|_)|dkDrtU||_+yt%j&|_+y)a Args: in_chs: Input channels. out_chs: Output channels. stride: Stride for conv. dilation: Dilation for conv. cfg: Convolution block configuration. drop_path: Drop path rate. rr)rCrDrt)r1x1rp)r*r*r*rr*r=r) apply_act)rCrt)r)rr;r.rtrvecarO rd_channelsNrr?),rr7rrrrQrOrUr#rjr;r5rmrrCrortrrrrqrnpre_normrrrr conv1_1x1rrl conv2_kxkrrwreryrar{rurse_earlyr"rv conv3_1x1rr)rYr9r:rr;rrnorm_act_layermid_chsr. stride_poolstride_1stride_2 dilation_2 attn_kwargsrs rZrzMbConvBlock.__init__sy$ k4)+ !3CNNCMM!RX[XdXde S->->'FcN^N^!^_CNNG4 Q;(3==sX[XcXceDMKKMDM"7777*1' Xx ??f $&,hqkK __ %#)8A;jH#)8A;jH&v9I9IJ ?$VVs?W?WadalalmDI DI&vw(K#G, & WcoojV cnnc *~~%5)@+.+=+= K(-0cN_N_7el1m-n M* >>'O;ODM'0DJDG DM'0DJ!#..'I[IDG&wQ09B),BKKMr\rrNc:ttt||yr&r.rr2r(s rZr)zMbConvBlock.init_weights\GJv6=r\rc|j|}|j|}|j|}|j|}|j |}|j |}|j |j |}|j|}|j|j|}|j|}|j||z}|SrW) rrArrBrrCrDr"rvrErrYrrs rZrzMbConvBlock.forward_s==# MM!  IIaL NN1  JJqM NN1  == $ a A JJqM 77  A NN1  NN1  (r\r+)r]r^r_r`r3rarrdrrer)rrrrrs@rZr7r7sQ (."0"2!FRFRFR FR CHo FR  FRFRP>3>>%,,r\r7ceZdZdZddddeddfded eed ed ed eeefd edede ffd Z de jde jfdZ xZS) ConvNeXtBlockzConvNeXt Block.Nr*r8Tr?r9r:rlrr;rconv_mlprc t||xs|}t|j} |r1t t |j |j} t} nd|j vsJt} t} ||_ |dk(rt|||_ nG||k7r)tj||d|j |_ ntj"|_ |j$dvsJd\} } |j$dk(r|} n|} | dk(rt|||j& |_ntj"|_t+|||| |dd |j |_| ||_| |t1|j2|z|j | |_|r<|j6rt9||j6ntj"|_n;|j6rt=||j6ntj"|_|d kDrt?||_ ytj"|_ y)ay Args: in_chs: Input channels. out_chs: Output channels. kernel_size: Kernel size for depthwise conv. stride: Stride for conv. dilation: Dilation for conv. cfg: Convolution block configuration. conv_mlp: Whether to use convolutional MLP. drop_path: Drop path rate. rrRrr*)rlrD)rrpr8rrT)rlrr; depthwiserD)rDrOr?N)!rrrrOrrrQrUrrr use_conv_mlprrrrrorrqrrrrconv_dwrrar;rrMrlsrrr)rYr9r:rlrr;rrUrrOrQ mlp_layerrH stride_dwrs rZrzConvNeXtBlock.__init__ys, #V!#--0  !?S\\RJI#..0 00"JI$ Q;(9DM w IIfg13??[DMKKMDM.000!% Y ??f $ KI ! $VVs?W?WXDI DI$ GYQYZ[Q\2 w' Wc#*:*:W*D&ECOOgpq @Cl7COOAooj#//:SUS^S^S`DG09B),BKKMr\rrc|j|}|j|}|j|}|jr4|j |}|j |}|j |}n[|jdddd}|j |}|j |}|j |}|jdddd}|j||z}|SNrrrkr*) rrrYrXrrrZpermuterrQs rZrzConvNeXtBlock.forwards==# IIaL LLO    ! A A A !Q1%A ! A A A !Q1%A NN1  (r\)r]r^r_r`r3rar rrcrdrrrrrrs@rZrSrSvs &* (."0"2!!?R?Rc]?R ?R  ?R CHo ?R ?R?R?RB%,,r\rSrrIcl|j\}}}}t||dzdk(d|d|ddt||dzdk(d|d|dd|j|||dz|d||dz|d|}|jddddd d j jd |d|d|}|S) z'Partition into non-overlapping windows.rheight () must be divisible by window ()r*width (rkrrrrr$rr_r)rrIrrrrwindowss rZwindow_partitionrhsJAq!Q A A ! #xs2QR]^_R`Qaab%cd A A ! #wqc1PQ\]^Q_P``a%bc q!{1~%{1~qKN7JKXYN\]^Aii1aAq)446;;B AP[\]P^`abG Nr\rgimg_sizec|\}}|jd}|jd||dz||dz|d|d|}|jddddddjjd|||}|S)zReverse window partition.rrr*rkrrerrrr_rrgrIrirrrrs rZwindow_reverserms DAq bA Rk!n,a;q>.A;q>S^_`SacdeA !Q1a#..055b!QBA Hr\rJc h|j\}}}}t||dzdk(d|d|dt||dzdk(d|d|d|j||d||dz|d||dz|}|jdddddd j jd |d|d|}|S) z6Partition into overlapping windows with grid striding.rheight  must be divisible by grid r*width rrerkrrrf)rrJrrrrrgs rZgrid_partitionrrsJAq!Q A !  !WQC/J9UV<.#YZ A !  !VA3.I)TU,#XY q)A,Yq\ 19Q<iPQlARTUVAii1aAq)446;;B ! iXYl\]^G Nr\c|\}}|jd}|jd||dz||dz|d|d|}|jddddddjjd|||}|S)zReverse grid partition.rrr*rkrerrrkrgrJrirrrrs rZ grid_reverserus DAq bA Ril*A1,=y|YWX\[\]A !Q1a#..055b!QBA Hr\rcd}|jdk(rtt||j}|S|jdk(rtt|}|S|jdk(rtt |}|S)z,Get relative position class based on config.Nr)rI hidden_dimrD)rIbias_tf)rErr%rGr&r')rrIrs rZget_rel_pos_clsrysuK 5 i[S__]   V #jkB    Y &l D r\c ReZdZdZdedfdedededeffd Zd Z d Z xZ S) PartitionAttentionClzRGrid or Block partition + Attn + FFN. NxC 'channels last' tensor layout. blockr?rpartition_typerrc t|tt|j|j }t |j}|dk(|_t|jr |jn |j|_ t||j}|||_t|||j |j"|j$||j&|j(|_|j,rt/||j,nt1j2|_|dkDr t7|nt1j2|_|||_t=|t?||j@z||j(|_!|j,rt/||j,nt1j2|_"|dkDrt7||_#yt1j2|_#y)Nrr|r8rDr9rr@rArr?r)$rrrrrSrUrrOpartition_blockr!rIrJpartition_sizeryrrr8r>r9r@rArrMrrrr rr!r"rrar;rr#r$ rYrr}rrrQrOrrs rZrzPartitionAttentionCl.__init__s| ^C,=,=>CLLQ !#--0 -8'4;O;OUXUbUbc%c4+>+>? _   \\~~#mmmm   DG??:cs?XZXcXcXe1:R(9-R[[]_ c&6&6 67   DG??:cs?XZXcXcXe1:R(9-R[[]r\c0|jdd}|jrt||j}nt ||j}|j |}|jrt ||j|}|St||j|}|S)Nr*rk)rrrhrrrrrmrurYrri partitioneds rZ_partition_attnz$PartitionAttentionCl._partition_attn-s771Q<   *1d.A.ABK(D,?,?@Kii ,   {D,?,?JA[$*=*=xHAr\c  ||j|j|j|j|z}||j |j |j |j|z}|SrWr!r rrr$r#rr"rs rZrzPartitionAttentionCl.forward<c )=)=djjm)L MN N $**Q-)@ AB Br\) r]r^r_r`r4rarerdrrrrrs@rZr{r{sQ#*)>)@! $S$S $S' $S  $SL r\r{ceZdZdZedfdededeffd Zdejdejfd Z dejdejfd Z xZ S) ParallelPartitionAttentionzQExperimental. Grid and Block partition + single FFN. NxC tensor layout. r?rrrc t||dzdk(sJtt|j|j }t |j}|j|jk(sJt|j|_ t||j}|||_ t||dz|j|j |j"||j$|j&|_t||dz|j|j |j"||j$|j&|_|j,rt/||j,nt1j2|_|dkDr t7|nt1j2|_|||_t=|t?||j@z|||j&|_!|j,rt/||j,nt1j2|_"|dkDrt7||_#yt1j2|_#y) z Args: dim: Input dimension. cfg: Transformer block configuration. drop_path: Drop path rate. rrrrrr?)rr out_featuresrOrN)$rrrrrSrUrrOrIrJr!rryrrr8r>r9r@rA attn_block attn_gridrMrrrr rr!r"rrar;rr#r$)rYrrrrQrOrrs rZrz#ParallelPartitionAttention.__init__Hs Qw!||^C,=,=>CLLQ !#--0 #--///'8%c4+>+>? _ %  1H\\~~#mmmm  %  1H\\~~#mmmm  DG??:cs?XZXcXcXe1:R(9-R[[]_ c&6&6 67   DG??:cs?XZXcXcXe1:R(9-R[[]r\rrcJ|jdd}t||j}|j|}t ||j|}t ||j}|j |}t||j|}tj||gdS)Nr*rkrr) rrhrrrmrrrrurcat)rYrripartitioned_blockx_windowpartitioned_gridx_grids rZrz*ParallelPartitionAttention._partition_attns771Q<,Q0C0CD OO,=>!"3T5H5H(S)!T-@-@A>>*:;.0C0CXNyy(F+44r\c  ||j|j|j|j|z}||j |j |j |j|z}|SrWrrs rZrz"ParallelPartitionAttention.forwardrr\) r]r^r_r`r4rardrrrrrrrs@rZrrBso*?)@! 5S5S'5S 5Sn 5 5%,, 5%,,r\rc l|j\}}}}t||dzdk(d|d|ddt||dzdk(d|d|dd|j||||dz|d||dz|d}|jddddd d j jd ||d|d}|S) z#Partition windows for NCHW tensors.rrarbrcr*rdrrerkrrrf)rrIrrrrrgs rZwindow_partition_nchwrsJAq!Q A A ! #xs2QR]^_R`Qaab%cd A A ! #wqc1PQ\]^Q_P``a%bc q!Q+a.(+a.!{1~:M{[\~^Aii1aAq)446;;B;q>S^_`SabG Nr\c |\}}|jd}|jd||dz||dz||d|d}|jddddddjjd|||}|S)z*Reverse window partition for NCHW tensors.r*rrrkrerrrkrls rZwindow_reverse_nchwrs DAq aA Rk!n,a;q>.A1kRSnVabcVdeA !Q1a#..055b!QBA Hr\c h|j\}}}}t||dzdk(d|d|dt||dzdk(d|d|d|j|||d||dz|d||dz}|jdddddd j jd ||d|d}|S) z Grid partition for NCHW tensors.rrorpr*rqrkrrrerrf)rrJrrrrrgs rZgrid_partition_nchwrsJAq!Q A !  !WQC/J9UV<.#YZ A !  !VA3.I)TU,#XY q!Yq\1 ! #4ilASTDUVAii1aAq)446;;B9Qr9r@rArrMrrrr rr!r"rrar;rr#r$rs rZrzPartitionAttention2d.__init__sz ^CNN;N !#--0 -8'4;O;OUXUbUbc%c4+>+>? _   \\~~#mmmm   FI__<AZ\ZeZeZg1:R(9-R[[]_ c&6&6 67   FI__<AZ\ZeZeZg1:R(9-R[[]r\rrc0|jdd}|jrt||j}nt ||j}|j |}|jrt ||j|}|St||j|}|S)Nr)rrrrrrrrrs rZrz$PartitionAttention2d._partition_attns7723<   /43F3FGK-a1D1DEKii ,   #K1D1DhOA"+t/B/BHMAr\c  ||j|j|j|j|z}||j |j |j |j|z}|SrWrrs rZrzPartitionAttention2d.forwardrr\)r]r^r_r`r4rarerdrrrrrrrs@rZrrs}#*)>)@! +S+S +S' +S  +SZ  %,, %,,r\rc heZdZdZdeedfdededededed ef fd Zd d Z d Z xZ S) MaxxVitBlockz;MaxVit conv, window partition + FFN , grid partition + FFN.r*r?rrrrrrc>t ||j|_|jdk(rt nt }|||||||_t|||}|jrtnt} |jrdn| di||_ | dddi||_ y)a^Initialize MaxxVitBlock. Args: dim: Input channel dimension. dim_out: Output channel dimension. stride: Stride for downsampling. conv_cfg: Configuration for convolutional blocks. transformer_cfg: Configuration for transformer blocks. drop_path: Drop path rate. r}rrrrrrNr}gridrf)rrrL nchw_attnrirSr7convdictrr{rKrr) rYrrrrrrconv_clsrLpartition_layerrs rZrzMaxxVitBlock.__init__s& (66$,$7$7:$E=;S'&hR[\ wOyQ 26...FZ"1"?"?$_EcWbEc(NN+Nr\c|j%ttt||jttt||jttt ||j yr&)rr.rr rr2rr(s rZr)zMaxxVitBlock.init_weights#sM ?? &  1&A4?? SG-f=t~~NGJv6 Br\c|j|}|js|jdddd}|j|j|}|j |}|js|jdddd}|Sr^)rrr_rrrs rZrzMaxxVitBlock.forward)sp IIaL~~ !Q1%A ?? &"A NN1 ~~ !Q1%Ar\r+) r]r^r_r`r3r4rardrr)rrrs@rZrrskE '5'75J5L!OOO O % O 3 OO<C r\rceZdZdZddeedfdedededed ed ed effd Zdd e ddfdZ de jde jfdZ xZS)ParallelMaxxVitBlockzYMaxVit block with parallel cat(window + grid), one FF. Experimental timm block. r*rr?rrrnum_convrrrc$t ||jdk(rtnt}|dkDr<||||||g} | |||||g|dz zz } t j | |_n|||||||_t||||_ y)aa Args: dim: Input dimension. dim_out: Output dimension. stride: Stride for first conv block. num_conv: Number of convolution blocks. conv_cfg: Convolution block configuration. transformer_cfg: Transformer block configuration. drop_path: Drop path rate. r}r*r)rrrN) rrrirSr7rrrrr) rYrrrrrrrrconvsrs rZrzParallelMaxxVitBlock.__init__=s( $,$7$7:$E=; a<c76xS\]^E hwXSTX`cdXde eE u-DI gf(V_`DI.7[de r\rrNcttt||jttt||j yr&)r.rr rr2rr(s rZr)z!ParallelMaxxVitBlock.init_weights\s-G-f=tyyIGJv6 Br\rc|j|}|jdddd}|j|}|jdddd}|Sr^)rr_rrs rZrzParallelMaxxVitBlock.forward`sI IIaL IIaAq ! IIaL IIaAq !r\r+)r]r^r_r`r3r4rardrrer)rrrrrs@rZrr7s'5'75J5L!fff f  f % f3ff>C3CC%,,r\rceZdZdZddddeedfdeded ed ed eeefd ee ee fd ededee e e fffd Z de jde jfdZxZS) MaxxVitStagezEMaxxVit stage consisting of mixed convolution and transformer blocks.rre)rrr?r9r:rdepth feat_size block_typesrrrc t|d|_t||}g} t |D]\} } | dk(r|nd} | dvsJ| dk(r0|j dk(rt nt}| |||| || | gz } ne| dk(r%t||}| t||| ||| |  gz } n;| d k(r| t||| ||| |  gz } n| d k(r| t||| ||| |  gz } |}tj| |_y )a Args: in_chs: Input channels. out_chs: Output channels. stride: Stride for first block. depth: Number of blocks in stage. feat_size: Feature map size. block_types: Block types ('C' for conv, 'T' for transformer, etc). transformer_cfg: Transformer block configuration. conv_cfg: Convolution block configuration. drop_path: Drop path rate(s). Frr*)rrMPMrr}rr)rrrrr)rrrrrN)rrgrad_checkpointingr" enumeraterirSr7ryrrrrrblocks)rYr9r:rrrrrrrrit block_striderrrs rZrzMaxxVitStage.__init__ksj0 "'";6 k*( DAq%&!V6L-- --Cx,4,?,?:,M=S^8' 'l c-oyI -' +''l c<'%$3'l d/'%$3'l FQ( RmmV, r\rrc|jr6tjjst |j |}|S|j |}|SrW)rrjit is_scriptingr/rrs rZrzMaxxVitStage.forwardsE  " "599+A+A+Ct{{A.A AAr\)r]r^r_r`r4r3rarrrerdr rrrrrrs@rZrrhsO )1255J5L'5'735F-F-F- F-  F- S#X F-sE#J/F-3F-%F-UDK/0F-P%,,r\rceZdZdZ ddededededededed effd Zdd ed d fdZ de jd e jfdZ xZ S)Stemz"Stem layer for feature extraction.r9r:rlrtrDrOrQrUc >t |t|ttfs t |}t t|||} |d|_d|_ t||d|d|||_ | |d|_ t|d|d|d|||_ y)ae Args: in_chs: Input channels. out_chs: Output channels. kernel_size: Kernel size for convolutions. padding: Padding mode. bias: Whether to use bias. act_layer: Activation layer. norm_layer: Normalization layer. norm_eps: Normalization epsilon. rrrr)rrtrDr*N)rrrlisttupler!rrr:rrconv1rconv2) rYr9r:rlrtrDrOrQrUrFrs rZrz Stem.__init__s, 'D%=1(G !3J !JPXYr{  "671:{1V]dhi #GAJ/ "71:wqz;qZahlm r\rrNc:ttt||yr&rNr(s rZr)zStem.init_weightsrOr\rcl|j|}|j|}|j|}|SrW)rrrrs rZrz Stem.forwards. JJqM JJqM JJqMr\)rkrsFrNr~rr+)r]r^r_r`rarercrdrr)rrrrrs@rZrrs, !#+" n n n n  n  n n n nD>3>>%,,r\rc|j|jsJ|S|d|jz|d|jzf}t|||}|S)z>Configure window size based on image size and partition ratio.rr*)rIrJ)rIrJrHr)rrirs rZcfg_window_sizersX "}}} a[C$7$77!H[H[9[[N #>^ LC Jr\kwargsc Vi}i}i}|jD]X\}}|jdr|||jdd<-|jdr|||jdd<T|||<Zt|ft|jfi|t|jfi|d|}|S)z-Overlay keyword arguments onto configuration. transformer_rsconv_)rr)items startswithrrr)rrtransformer_kwargs conv_kwargs base_kwargsrrs rZ_overlay_kwargsrsKK 1 << '@A qyy< = \\' "23K '2. /KN     3 3J7IJ55    C Jr\cveZdZdZ d'dedeeeeeffdededede de d e ffd Z d(d e jd ed eddfdZej j"deefdZej j"d)dedeee ffdZej j"d*deddfdZej j"de jfdZd+dedeeddfdZ d,dej6deeeeefdededededeeej6eej6eej6fffdZ d-deeeefded edeed!ffd"Zdej6dej6fd#Zd)dej6d$edej6fd%Z dej6dej6fd&Z!xZ"S).r5z{CoaTNet + MaxVit base model. Highly configurable for different block compositions, tensor layouts, pooling types. rriin_chans num_classes global_pool drop_ratedrop_path_raterc >t|t|}|r t|fi|}t |j |} ||_||_|jdx|_ |_||_ d|_ g|_ t||j|jj |j"|jj$|jj&|jj(|_|j*j,} |xjt/|j*j0ddgz c_ t3t5|t| D cgc] \} } | | z c} } } t7|j}t7|j8|k(sJt;j<d|t?|j8jA|j8Dcgc]}|jC}}|j*j0}g}tE|D]} d}|j| }t3| Dcgc] }|dz |zdzc}} |tG|||j8| |jH| |j| | ||  gz }| |z} |}|xjt/|| d | gz c_ tKjL||_'tQtS|j j&|j j( }|jTrUtKjV|_,|jT|_*t[|j||jT||| |_.nF|j|_*||j|_,t_|j||| |_.|j`dvsJ|j`r,tctQ|jd|j`|yycc} } wcc}wcc}w)a Args: cfg: Model configuration. img_size: Input image size. in_chans: Number of input channels. num_classes: Number of classification classes. global_pool: Global pooling type. drop_rate: Dropout rate. drop_path_rate: Drop path rate. **kwargs: Additional keyword arguments to overlay on config. rF)r9r:rtrDrOrQrUrstem)num_chs reductionrrr*)rrrrrrzstages.r) hidden_sizerCrrQ)rCr)rsrrrrr'N)3rrr!rrrrrr num_featuresrr feature_inforrrrtrrOrQrUrrrr:rziplenrrlinspacesumsplittolistrangerrirrstagesrrrrrrheadrrr. _init_weights)rYrrirrrrrrrrrsr num_stagesrdprr9r stage_strider:rfinal_norm_layerrs rZrzMaxxVit.__init__s, X& !#00C)#*=*=xH&&-0]]2->>DN""'NNLL((ll,,||..\\**  !! d499+<+<RXYZZc(If>!^S_#U#[#[\_\f\f#ghaqxxzhh""z" aALmmA&GINqA,6:NOI |jjmNN1- /#a&   F l "FF   $w&SZ[\Z]Q^"_!` ` ! a"mmV, ">#2E2E2P2P#QWZWjWjWsWst    DI$'$8$8D !-!! 11%#+ DI%)$5$5D !():):;DI&t'8'8+Q\hqrDI"\\\\ ??  2 23??KT R ]Pi Os!P ,PP rrrrNct|dr |j|yy#t$r|jYywxYw)Nr)r')hasattrr) TypeError)rYrrrs rZrzMaxxVit._init_weightsmsD 6> * &##6#2 + &##% &s ">>c|jDchc]\}tfddDrc}}Scc}}w)Nc3&K|]}|v ywrWrf).0nrs rZ z*MaxxVit.no_weight_decay..xsSa16Ss)relative_position_bias_tablez rel_pos.mlp)named_parametersany)rYr_s ` rZno_weight_decayzMaxxVit.no_weight_decaytsK//1UU!QS#RSS U UUs :coarsec$tdddg}|S)Nz^stem)z^stages\.(\d+)N)z^norm)i)rr)r)rYr matchers rZ group_matcherzMaxxVit.group_matcherzs-/CD r\enablec4|jD] }||_ yrW)rr)rYr rs rZset_grad_checkpointingzMaxxVit.set_grad_checkpointings *A#)A  *r\c.|jjSrW)rfcrXs rZget_classifierzMaxxVit.get_classifiersyy||r\cJ||_|jj||yrW)rrreset)rYrrs rZreset_classifierzMaxxVit.reset_classifiers&  [1r\rindicesr stop_early output_fmtintermediates_onlyc|dvsJdg}tt|jdz|\}} d} |j|}| |vr|j |t|j} t j js|s |j} n|jd| } | D]@} | dz } | |}| |vs|r| | k(r|j|}n|}|j |B|r|S| | k(r|j|}||fS)a Forward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if int, all if None, select matching indices if sequence norm: Apply norm layer to compatible intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features Returns: )NCHWzOutput shape must be NCHW.r*rN) r,rrrappendrrrr)rYrrrrrr intermediates take_indices max_indexfeat_idxlast_idxrstagex_inters rZforward_intermediateszMaxxVit.forward_intermediatess*Y&D(DD& "6s4;;7G!7KW"U i IIaL | #   #t{{# 99 ! ! #:[[F[[),F .E MHaA<'H0"iilGG$$W- .  x  ! A-r\ prune_norm prune_head.ctt|jdz|\}}|jd||_|rtj|_|r|j dd|_|S)z6Prune layers not required for specified intermediates.r*Nrrs)r,rrrrrrr)rYrr%r&rrs rZprune_intermediate_layersz!MaxxVit.prune_intermediate_layerssb#7s4;;7G!7KW"U ikk*9-  DI --a4DIr\cl|j|}|j|}|j|}|SrW)rrrrs rZforward_featureszMaxxVit.forward_featuress. IIaL KKN IIaLr\ pre_logitscN|r|j||S|j|S)N)r+)r)rYrr+s rZ forward_headzMaxxVit.forward_heads%6@tyyzy2RdiiPQlRr\cJ|j|}|j|}|SrW)r*r-rs rZrzMaxxVit.forwards'  ! !! $   a r\)rkrr?r?r+F)TrW)NFFrF)r*FT)#r]r^r_r`r2rrarrerdrrrModulerrrignorer rrcr r rrr rrr r$r(r*r-rrrs@rZr5r5 s58#$!$&\S\SCsCx01\S \S  \S  \S\S"\S\S|&BII&S&#&t& YYUSUU  YYDT#s(^ YY*T*T** YY 2C2hsm2W[28<$$',4 ||4 eCcN344  4  4  4 !%4  tELL!5tELL7I)I#JJ K4 p./$# 3S >*    sCx %,,5<< SellSSS%,,r\r5rFTrDrFrqrCconv_output_biasconv_attn_earlyconv_attn_act_layerconv_norm_layertransformer_shortcut_biastransformer_norm_layertransformer_norm_layer_clrMrErGc btt||dd|||d| td||| ||| | S)asRW variant configuration for CoAtNet models. These models were created and trained before seeing https://github.com/google-research/maxvit Common differences for initial timm models: - pre-norm layer in MZBConv included an activation after norm - mbconv expansion calculated from input instead of output chs - mbconv shortcut and final 1x1 conv did not have a bias - SE act layer was relu, not silu - mbconv uses silu in timm, not gelu - expansion in attention block done via output proj, not input proj Variable differences (evolved over training initial models): - avg pool with kernel_size=2 favoured downsampling (instead of maxpool for coat) - SE attention was between conv2 and norm/act - default to avg pool for mbconv downsample instead of 1x1 or dw conv - transformer block shortcut has no bias TFrx) rqrCrnrjroruryrOrQ)r<r=rCrMrQrSrErGrrrr3r4) rqrCr4r5r6r7r8r9r:rMrErGs rZ _rw_coat_cfgr>sW@ #(&.&  .3#-3%#   r\rpconv_attn_ratior8c `tt||d||d|td|||| ||| |  S)aRW variant configuration for MaxViT models. These models were created and trained before seeing https://github.com/google-research/maxvit Differences of initial timm models: - mbconv expansion calculated from input instead of output chs - mbconv shortcut and final 1x1 conv did not have a bias - mbconv uses silu in timm, not gelu - expansion in attention block done via output proj, not input proj Frx)rqrCrjror{rOrQ) r<rCr8rIrMrQrSrErGr<r=) rqrCr4r?r7r9r:rIr8rMrErGs rZ _rw_max_cfgrAsS0 #(&& .##-3%#   r\rTrconv_norm_layer_clrKc t|}ttd||d|d||td||||d||| |  S)z=Configuration for experimental ConvNeXt-based MaxxViT models.r}Fr)rirqrCrjrMrQrSr*) r<rCrIrKrMrQrSrErGr<)r!rr3r4) rqrCr7rBr9r:rIrKrMrErGs rZ _next_cfgrDLseK(K !##A&, .#'#A-3%#   r\c Nttdddtdddd S) z0Configuration matching TensorFlow MaxViT models.gMbP? gelu_tanhsame)rUrOrtrFrx)rUrOr9rEr<r=rfr\rZ_tf_cfgrHss6 ! .!"   r\coatnet_pico_rw)rrFr)r7r)rrrrz)r4r?coatnet_nano_rw)rkrerk)rqr4r? coatnet_0_rwr)rrkrTr)r5r8 coatnet_1_rw)rrMrr)rqr5r8 coatnet_2_rw)rJrKrF)rrJrx)rqr6 coatnet_3_rw)rrr)rr)rqr6rMcoatnet_bn_0_rwr~)rqr5r8r9coatnet_rmlp_nano_rwr)r4r?rErGcoatnet_rmlp_0_rw)rqrEcoatnet_rmlp_1_rwr)rCr5r8rErGcoatnet_rmlp_1_rw2)rqrErGcoatnet_rmlp_2_rw)rqr6rMrEcoatnet_rmlp_3_rwcoatnet_nano_cc)rrrrr\)rrrricoatnext_nano_rwr)rrrr)rN)rErM coatnet_0rr)rrrr coatnet_1 coatnet_2rJrQ coatnet_3rrS coatnet_4)r r coatnet_5)rKrFrgmaxvit_pico_rw)r7rrJrK)rrrr)rrrr)r7)rrrirmaxvit_nano_rw)r*rrkr*maxvit_tiny_rwmaxvit_tiny_pm)rrrrmaxvit_rmlp_pico_rw)rEmaxvit_rmlp_nano_rwmaxvit_rmlp_tiny_rwmaxvit_rmlp_small_rwmaxvit_rmlp_base_rw)rrrirrmaxxvit_rmlp_nano_rw)rrrirrmaxxvit_rmlp_tiny_rwmaxxvit_rmlp_small_rw)0rmaxxvitv2_nano_rw)rKrEmaxxvitv2_rmlp_base_rw)rrMrcr)rKmaxxvitv2_rmlp_large_rw)i@irf)rrMr)Pryrfmaxvit_tiny_tf)rrrirrrmaxvit_small_tfmaxvit_base_tfmaxvit_large_tfmaxvit_xlarge_tf state_dictmodelcr|j}i}|jD]\}}|jdr|j|dd}|j|j jk7s|j d|j dk7r,t||j |j j}||vrn|j||jk7rR|j||jk(r.|jdvsJ|j||j}|||<|S)z/Filter checkpoint state dict for compatibility.rNirr*)new_window_sizenew_bias_shape)rre) rrendswith get_submodulerrrIr)ndimnumelr)rrmodel_state_dictout_dictrrms rZcheckpoint_filter_fnrs)'')H  "1 ::4 5##AdsG,Aww!88>>>!--PQBRVWVcVcdeVfBf-$%MM#$#A#A#G#G  QVV/?/B/G/G%GAGGIYijkYlYrYrYtLt66V# ## *1-334A  Or\variant cfg_variant pretrainedc |.|tvr|}n#dj|jddd}tt||ft|t dt d|S)zCreate a MaxxVit model variant.NrrT)flatten_sequential) model_cfg feature_cfgpretrained_filter_fn) model_cfgsjoinrr+r5rr)rrrrs rZ_create_maxxvitr si j !K((7==#5cr#:;K * [)D11     r\urlc "|dddddddddd d |S) z$Create a default configuration dict.r0)rkr/r/)rTrTffffff?bicubic)?rrz stem.conv1zhead.fcT) rr input_size pool_sizecrop_pct interpolationmeanr first_conv classifierfixed_input_sizerf)rrs rZ_cfgrs44}SY9")     r\zcoatnet_pico_rw_224.untrained)rzcoatnet_nano_rw_224.sw_in1kztimm/zyhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/coatnet_nano_rw_224_sw-f53093b4.pthg?) hf_hub_idrrzcoatnet_0_rw_224.sw_in1kzvhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/coatnet_0_rw_224_sw-a6439706.pth)rrzcoatnet_1_rw_224.sw_in1kzvhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/coatnet_1_rw_224_sw-5cae1ea8.pthz!coatnet_2_rw_224.sw_in12k_ft_in1k)rz'coatnet_rmlp_1_rw2_224.sw_in12k_ft_in1kz&coatnet_rmlp_2_rw_224.sw_in12k_ft_in1kz&coatnet_rmlp_2_rw_384.sw_in12k_ft_in1k)rkrr)rcrcg?squash)rrrr crop_modezcoatnet_bn_0_rw_224.sw_in1kzyhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/coatnet_bn_0_rw_224_sw-c228e218.pthr)rrrrrz coatnet_rmlp_nano_rw_224.sw_in1kz~https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/coatnet_rmlp_nano_rw_224_sw-bd1d51b3.pthzcoatnet_rmlp_0_rw_224.untrainedzcoatnet_rmlp_1_rw_224.sw_in1kz{https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/coatnet_rmlp_1_rw_224_sw-9051e6c3.pthzcoatnet_rmlp_2_rw_224.sw_in1kz{https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/coatnet_rmlp_2_rw_224_sw-5ccfac55.pthzcoatnet_rmlp_3_rw_224.untrainedzcoatnet_nano_cc_224.untrainedzcoatnext_nano_rw_224.sw_in1kzzhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/coatnext_nano_rw_224_ad-22cb71c2.pthzcoatnet_2_rw_224.sw_in12ki-.)rrzcoatnet_3_rw_224.sw_in12kzcoatnet_rmlp_1_rw2_224.sw_in12kzcoatnet_rmlp_2_rw_224.sw_in12kzcoatnet_0_224.untrainedzcoatnet_1_224.untrainedzcoatnet_2_224.untrainedzcoatnet_3_224.untrainedzcoatnet_4_224.untrainedzcoatnet_5_224.untrainedzmaxvit_pico_rw_256.untrained)rkrKrK)r)rrrzmaxvit_nano_rw_256.sw_in1kzxhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/maxvit_nano_rw_256_sw-fb127241.pth)rrrrzmaxvit_tiny_rw_224.sw_in1kzxhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/maxvit_tiny_rw_224_sw-7d0dffeb.pthzmaxvit_tiny_rw_256.untrainedzmaxvit_tiny_pm_256.untrainedzmaxvit_rmlp_pico_rw_256.sw_in1kz}https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/maxvit_rmlp_pico_rw_256_sw-8d82f2c6.pthzmaxvit_rmlp_nano_rw_256.sw_in1kz}https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/maxvit_rmlp_nano_rw_256_sw-c17bb0d6.pthzmaxvit_rmlp_tiny_rw_256.sw_in1kz}https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/maxvit_rmlp_tiny_rw_256_sw-bbef0ff5.pthz maxvit_rmlp_small_rw_224.sw_in1kz~https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/maxvit_rmlp_small_rw_224_sw-6ef0ae4f.pthz"maxvit_rmlp_small_rw_256.untrainedz(maxvit_rmlp_base_rw_224.sw_in12k_ft_in1kz(maxvit_rmlp_base_rw_384.sw_in12k_ft_in1kz maxvit_rmlp_base_rw_224.sw_in12kz maxxvit_rmlp_nano_rw_256.sw_in1kz~https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/maxxvit_rmlp_nano_rw_256_sw-0325d459.pthz"maxxvit_rmlp_tiny_rw_256.untrainedz!maxxvit_rmlp_small_rw_256.sw_in1kzhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights-maxx/maxxvit_rmlp_small_rw_256_sw-37e217ff.pthzmaxxvitv2_nano_rw_256.sw_in1k)rrrz+maxxvitv2_rmlp_base_rw_224.sw_in12k_ft_in1kz+maxxvitv2_rmlp_base_rw_384.sw_in12k_ft_in1kz%maxxvitv2_rmlp_large_rw_224.untrainedz#maxxvitv2_rmlp_base_rw_224.sw_in12kzmaxvit_tiny_tf_224.in1k)rrrzmaxvit_tiny_tf_384.in1kzmaxvit_tiny_tf_512.in1k)rkrFrF)rzrzzmaxvit_small_tf_224.in1kzmaxvit_small_tf_384.in1kzmaxvit_small_tf_512.in1kzmaxvit_base_tf_224.in1kzmaxvit_base_tf_384.in1kzmaxvit_base_tf_512.in1kzmaxvit_large_tf_224.in1kzmaxvit_large_tf_384.in1kzmaxvit_large_tf_512.in1kzmaxvit_base_tf_224.in21kiSUz maxvit_base_tf_384.in21k_ft_in1kz maxvit_base_tf_512.in21k_ft_in1kzmaxvit_large_tf_224.in21kz!maxvit_large_tf_384.in21k_ft_in1kz!maxvit_large_tf_512.in21k_ft_in1k)rrrrzmaxvit_xlarge_tf_224.in21kz"maxvit_xlarge_tf_384.in21k_ft_in1kz"maxvit_xlarge_tf_512.in21k_ft_in1kc tdd|i|S)z)CoatNet Pico model with RW configuration.r)coatnet_pico_rw_224rrrs rZrr   RZ R6 RRr\c tdd|i|S)z)CoatNet Nano model with RW configuration.r)coatnet_nano_rw_224rrs rZrr rr\c tdd|i|S)z&CoatNet-0 model with RW configuration.r)coatnet_0_rw_224rrs rZrr   O* O OOr\c tdd|i|S)z&CoatNet-1 model with RW configuration.r)coatnet_1_rw_224rrs rZrr rr\c tdd|i|S)z&CoatNet-2 model with RW configuration.r)coatnet_2_rw_224rrs rZrr rr\c tdd|i|S)z&CoatNet-3 model with RW configuration.r)coatnet_3_rw_224rrs rZrr rr\c tdd|i|S)z4CoatNet-0 model with BatchNorm and RW configuration.r)coatnet_bn_0_rw_224rrs rZrr% rr\c tdd|i|S)z.CoatNet Nano model with Relative Position MLP.r)coatnet_rmlp_nano_rw_224rrs rZrr+   W* WPV WWr\c tdd|i|S)z+CoatNet-0 model with Relative Position MLP.r)coatnet_rmlp_0_rw_224rrs rZrr1   Tz TV TTr\c tdd|i|S)z+CoatNet-1 model with Relative Position MLP.r)coatnet_rmlp_1_rw_224rrs rZrr7 rr\c tdd|i|S)z.CoatNet-1 model with Relative Position MLP v2.r)coatnet_rmlp_1_rw2_224rrs rZrr= s  U Uf UUr\c tdd|i|S)z+CoatNet-2 model with Relative Position MLP.r)coatnet_rmlp_2_rw_224rrs rZrrC rr\c tdd|i|S)z6CoatNet-2 model with Relative Position MLP at 384x384.r)coatnet_rmlp_2_rw_384rrs rZrrI rr\c tdd|i|S)z+CoatNet-3 model with Relative Position MLP.r)coatnet_rmlp_3_rw_224rrs rZrrO rr\c tdd|i|S)z(CoatNet Nano model with ConvNeXt blocks.r)coatnet_nano_cc_224rrs rZrrU rr\c tdd|i|S)z*CoAtNeXt Nano model with RW configuration.r)coatnext_nano_rw_224rrs rZrr[ s  Sj SF SSr\c tdd|i|S)zCoatNet-0 model.r) coatnet_0_224rrs rZrra   Lz LV LLr\c tdd|i|S)zCoatNet-1 model.r) coatnet_1_224rrs rZrrg rr\c tdd|i|S)zCoatNet-2 model.r) coatnet_2_224rrs rZrrm rr\c tdd|i|S)zCoatNet-3 model.r) coatnet_3_224rrs rZrrs rr\c tdd|i|S)zCoatNet-4 model.r) coatnet_4_224rrs rZrry rr\c tdd|i|S)zCoatNet-5 model.r) coatnet_5_224rrs rZrr rr\c tdd|i|S)z(MaxViT Pico model with RW configuration.r)maxvit_pico_rw_256rrs rZrr   QJ Q& QQr\c tdd|i|S)z(MaxViT Nano model with RW configuration.r)maxvit_nano_rw_256rrs rZrr rr\c tdd|i|S)z(MaxViT Tiny model with RW configuration.r)maxvit_tiny_rw_224rrs rZrr rr\c tdd|i|S)z3MaxViT Tiny model with RW configuration at 256x256.r)maxvit_tiny_rw_256rrs rZrr rr\c tdd|i|S)z3MaxViT Relative Position MLP Pico RW 256x256 model.r)maxvit_rmlp_pico_rw_256rrs rZrr   V Vv VVr\c tdd|i|S)z3MaxViT Relative Position MLP Nano RW 256x256 model.r)maxvit_rmlp_nano_rw_256rrs rZrr rr\c tdd|i|S)z3MaxViT Relative Position MLP Tiny RW 256x256 model.r)maxvit_rmlp_tiny_rw_256rrs rZrr rr\c tdd|i|S)z4MaxViT Relative Position MLP Small RW 224x224 model.r)maxvit_rmlp_small_rw_224rrs rZrr rr\c tdd|i|S)z9MaxViT Small model with Relative Position MLP at 256x256.r)maxvit_rmlp_small_rw_256rrs rZrr rr\c tdd|i|S)z-MaxViT Base model with Relative Position MLP.r)maxvit_rmlp_base_rw_224rrs rZrr rr\c tdd|i|S)z8MaxViT Base model with Relative Position MLP at 384x384.r)maxvit_rmlp_base_rw_384rrs rZrr rr\c tdd|i|S)z'MaxViT Tiny model with parallel blocks.r)maxvit_tiny_pm_256rrs rZrr rr\c tdd|i|S)z4MaxxViT Relative Position MLP Nano RW 256x256 model.r)maxxvit_rmlp_nano_rw_256rrs rZrr rr\c tdd|i|S)z.MaxxViT Tiny model with Relative Position MLP.r)maxxvit_rmlp_tiny_rw_256rrs rZrr rr\c tdd|i|S)z/MaxxViT Small model with Relative Position MLP.r)maxxvit_rmlp_small_rw_256rrs rZrr s  X: XQW XXr\c tdd|i|S)zMaxxViT-V2 Nano model.r)maxxvitv2_nano_rw_256rrs rZrr rr\c tdd|i|S)z1MaxxViT-V2 Base model with Relative Position MLP.r)maxxvitv2_rmlp_base_rw_224rrs rZrr   YJ YRX YYr\c tdd|i|S)zrArDrHrrrrr default_cfgsrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr r rrrrrrrrr r"r%r'rfr\rZr7s "H #11III A``ttZZff*+34< N 22 2B !P!P !PH  ! !  !N"))NbO"))Od<<&<299<&-299-`0bii0s0C002HHV,ryy,,S,$,0&8C=&C&C&b"))bJTBIITn 49 ELL tCy DQTI Z_ZfZf  elltCyU\\ %,, 49 S  V[VbVb   . U38_ QYZbQc >299>BMM`U\\S ell  DI QUVYQZ _d_k_k  5<<DI%,, u|| S  TRUY [`[g[g  E299EP22992j.299.bP299Pf,299,^.%S/Ncsz*WbiiWv"!& %#)!*.&3)4'+"6666 6 ! 6  6$(6!$6$'6e_666 #s(^6t !&!'!&3)415'+"---- -  - !$ -$'-eCHo.--e_--- #s(^-b ,"-&3)415#9=!$$$$ $ !$ $ $' $eCHo.$$5%u "556$$$ #s(^$Nc3h"n% !  n %  !  n*%  &+ +n< %   &+ =nP '   & Qnd '   & en| %   &+#0  }nR$ %  !   Snh!%  inz! %   &+  {nR" %   Snf! '   &  gn|! '   &  }nT%5  . Unb %  $ cnz% {nF% GnR' Sn^' _nj' knv( wnF$  - GnT%  - Unb%  - cnp%  - qn@#$  5 ) AnN#%  5 ) On\#%  5 ) ]nj$ %   kn~# %   nT$%  + Und$%  + enr%%  + snB ! %   C nX &'   Y nj ' '   k nB %  )C nT %  )U nf %  )g nx '  )y nJ  '  )K n b T#u||*;%<RYYSWX[]b]i]iXiSj, S x} QU il qx  c # $sCx. %X&#Tb\X&"4 H$X& E!FX& E!X& (*!X&*.t0+X&.-d//X&2-d Hsh/X3X&<"4 H "(< $=X&F' M)GX&N&t|OX&P$T J&KQX&V$T J&KWX&\&t|]X&^$Tb\_X&`#D I%aX&l "mX&r "sX&x&t(yX&~%d'X&Ht|IX&Jt|KX&Lt|MX&Nt|OX&Pt|QX&Rt|SX&X#DRMU[$\YX&Z!$ G F#4[X&b!$ G#HcX&h#D  F%4iX&n#DRMU[$\oX&t&t L F(4uX&|&t L F(4}X&D&t L F(4EX&L' M)MX&V)$  F+4WX&`/1aX&f/ Hsh1XgX&p')qX&|' M F)4}X&D)$2-[a*bEX&F( N F*4GX&R$T F&4SX&X244YX&\24 Hsh4X]X&b,Tb\cX&f*4,gX&pt "(< >qX&vt Hsh XwX&|t Hsh X}X&B "(<!>CX&H Hsh!XIX&N Hsh!XOX&Tt "(< >UX&Zt Hsh X[X&`t Hsh XaX&f "(<!>gX&l Hsh!XmX&r Hsh!XsX&z!{X&@' Hsh)XAX&F' Hsh)XGX&L "MX&R( Hsh*XSX&X( 3(*DYX&^!$#_X&d)$ Hsh+XeX&j)$ Hsh+XkX&X vSDSCSGSS SDSCSGSS PPPPP PPPPP PPPPP PPPPP SDSCSGSS XXXXX UdUcUgUU UdUcUgUU VtVsVwVV UdUcUgUU UdUcUgUU UdUcUgUU SDSCSGSS TTTSTWTT MdMcMgMM MdMcMgMM MdMcMgMM MdMcMgMM MdMcMgMM MdMcMgMM R4R3R7RR R4R3R7RR R4R3R7RR R4R3R7RR WWWWW WWWWW WWWWW XXXXX XXXXX WWWWW WWWWW R4R3R7RR XXXXX XXXXX Y$Y#Y'YY UdUcUgUU Z4Z3Z7ZZ Z4Z3Z7ZZ [D[C[G[[ d4d3d7dd d4d3d7dd d4d3d7dd fDfCfGff fDfCfGff fDfCfGff d4d3d7dd d4d3d7dd d4d3d7dd fDfCfGff fDfCfGff fDfCfGff hThShWhh hThShWhh hThShWhhr\