khCdZddlZddlmZddlmZddlmZmZddl Z ddl m Z ddl m Z mZmZddlmZmZmZmZmZdd lmZdd lmZdd lmZmZd gZd*d e j<dede fdZ!Gdde j<Z"Gdde j<Z#Gdde#Z$Gdde j<Z%Gdde j<Z&Gdde j<Z'Gdde j<Z(Gdd e j<Z)d Z*d+d!Z+d,d"Z,ee,d#$e,d#$e,d#$d%Z-ed+d&e)fd'Z.ed+d&e)fd(Z/ed+d&e)fd)Z0y)-zk Sequencer Paper: `Sequencer: Deep LSTM for Image Classification` - https://arxiv.org/pdf/2205.01972.pdf N)partial) accumulate)OptionalTuple)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STDDEFAULT_CROP_PCT) lecun_normal_DropPathMlp PatchEmbedClassifierHead)build_model_with_cfg) named_apply)register_modelgenerate_default_cfgs Sequencer2dmodulename head_biasct|tjrC|jdrTtjj |j tjj|j|y|rKt|j |jtjj |jytjj|j |jZd|vr,tjj|jdytjj |jyyyt|tjrLt|j |j*tjj |jyyt|tjtjtjfrStjj!|j tjj |jyt|tj"tj$tj&frZdt)j*|j,z }|j/D]$}tjj1|| |&yt3|dr|j5yy)Nheadmlpư>)stdg? init_weights) isinstancennLinear startswithinitzeros_weight constant_biasr xavier_uniform_normal_Conv2d LayerNorm BatchNorm2d GroupNormones_RNNGRULSTMmathsqrt hidden_size parametersuniform_hasattrr)rrrflaxstdvr$s Q/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/models/sequencer.py _init_weightsr:s&"))$ ??6 " GGNN6== ) GG  fkk9 5fmm,;;*GGNN6;;/'' 6;;*} >v{{3 + + FBII &fmm$ ;; " GGNN6;; ' # FR\\2>>2<<H I  fmm$ v{{# FRVVRVVRWW5 6TYYv1122'') 2F GG  VdUD 1 2  ( )c`eZdZfdZdej deej dffdZxZS) RNNIdentityc*tt| yN)superr=__init__)selfargskwargs __class__s r9rAzRNNIdentity.__init__<s k4)+r;xreturnNc |dfSr?rBrFs r9forwardzRNNIdentity.forward?s $wr;) __name__ __module__ __qualname__rAtorchTensorrrK __classcell__rEs@r9r=r=;s-,% d0B*Cr;r=c FeZdZ ddededededef fd ZdZxZS) RNN2dBase input_sizer3 num_layersr& bidirectionalcHt|||_||_|rd|zn||_||_d|_d|_||_d|_ |r|dk(r*tjd|jz||_ n|dk(r'tj|j||_ nY|dk(r.tj|j||_ d|_n&|dk(r-tj|j||_ d|_ntd|z|dk(r2 d|jz|k7rtd d|jzd |d |dk(r, |j|k7rtd |jd |d |dk(r3|j|k7rtd |jd |d d|_nF|dk(r3|j|k7rtd |jd |d d|_ntd|zt|_t|_y) NTcataddverticalF horizontalzUnrecognized union: zThe output channel z% is different from the input channel .)r@rArUr3 output_sizeunion with_verticalwith_horizontalwith_fcfcrr ValueErrorr=rnn_vrnn_h rBrUr3rVr&rWr`rcrEs r9rAzRNN2dBase.__init__Es2 $&.;1{? !#  ~))A(8(8$8*E%))D$4$4jA*$))D$4$4jA',$,&))D$4$4jA%*" !7%!?@@ e^ 4###z1 #6q4;K;K7K6LLqr|q}}~!AA e^ :- #6t7G7G6HHmnxmyyz!{|| j :- #6t7G7G6HHmnxmyyz!{||#(D l ":- #6t7G7G6HHmnxmyyz!{||!&D 3e;< < ]  ] r;cR|j\}}}}|jrd|jdddd}|jd||}|j |\}}|j|||d}|jdddd}nd}|j r<|jd||}|j |\}}|j|||d}nd}|1|/|jdk(rtj||gd}n||z}n ||}n||}|j|j|}|S)NrrYrrZ)dim) shaperapermutereshaperfrbrgr`rOrZrd) rBrFBHWCv_hs r9rKzRNN2dBase.forward~s'WW 1a    !Q1%A "a#A::a=DAq !Q2&A !Q1%AA    "a#A::a=DAq !Q2&AA =Q]zzU"IIq!f"-E ]A ]A 77  Ar;rTTrZT)rLrMrNintboolrArKrQrRs@r9rTrTCsO  "&7#7#7# 7#  7#  7#r r;rTc @eZdZ ddededededef fd ZxZS)LSTM2drUr3rVr&rWc t|||||||||jr tj|||d|||_|j r!tj|||d|||_yy)NT) batch_firstr&rW)r@rArarr0rfrbrgrhs r9rAzLSTM2d.__init__s} [*dMSXZab    + DJ    + DJ r;rw)rLrMrNrxryrArQrRs@r9r{r{sN  "&     r;r{c teZdZdeeeejdejddddddf fd Z d Z xZ S) Sequencer2dBlock@repsrTrZc t|t||z}|||_||||| | | |_| dkDr t | nt j|_|||_ ||||| |_ y)N)rVrWr`rcr) act_layerdrop) r@rArxnorm1 rnn_tokensr rIdentity drop_pathnorm2 mlp_channels)rBrlr3 mlp_ratio rnn_layer mlp_layer norm_layerrrVrWr`rcrr channels_dimrEs r9rAzSequencer2dBlock.__init__s 9s?+ _ #  !'  1:B),BKKM_ %c<9SWXr;c||j|j|j|z}||j|j|j |z}|Sr?)rrrrrrJs r9rKzSequencer2dBlock.forwardsQ ttzz!}=> > t00A?@ @r;) rLrMrNr{r rrr*GELUrArKrQrRs@r9rrs@ r||6ggY>r;rc$eZdZfdZdZxZS)Shufflec"t|yr?)r@rA)rBrEs r9rAzShuffle.__init__s r;c|jrZ|j\}}}}tj||z}|j |d|}|dd|ddfj |||d}|S)Nrk)trainingrmrOrandpermro)rBrFrprqrrrsrs r9rKzShuffle.forwardsc ==JAq!Qq1u%A !R#A!Q' ""1aB/Ar;rLrMrNrArKrQrRs@r9rrs r;rc$eZdZfdZdZxZS) Downsample2dc^t|tj|||||_y)N) kernel_sizestride)r@rArr)down)rB input_dim output_dim patch_sizerEs r9rAzDownsample2d.__init__s% IIiT^_ r;cx|jdddd}|j|}|jdddd}|S)NrrjrrY)rnrrJs r9rKzDownsample2d.forwards< IIaAq ! IIaL IIaAq !r;rrRs@r9rrs`r;rc veZdZdeeeeejdejddddddf fd Z d Z xZ S) Sequencer2dStageFrrrTrZrc\t||rt||||_n ||k(sJt j |_g}t |D]A}|j||||| | | | | ||||t|ttfr||n| Ct j||_ y)N) rrrrrrVrWr`rcrr) r@rAr downsamplerrrangeappendrlisttuple Sequentialblocks)rBrldim_outdepthrr3rr block_layerrrrrrVrWr`rcrrr block_idxrEs r9rAzSequencer2dStage.__init__s*  *3DDO'> !> kkmDOu I MM+###%#%+2z&Sequencer2d.__init__..es q1ur;rrrr3rrrrrrrVrWr`rcrrstages.)num_chs reductionr) pool_type drop_rate input_fmt)nlhb)r@rA num_classes global_pool num_featureshead_hidden_size feature_dimr feature_infor stemlenrr enumeraterdictrrstagesnormrrr)rBrrrrlayers patch_sizes embed_dims hidden_sizes mlp_ratiosrrrrrnum_rnn_layersrWr`rcrdrop_path_rater stem_norm reductionsrprev_dimirurEs r9rAzSequencer2d.__init__8sc2 k)))&&4>rNBD1 "1~ m%.zD  6{c+.i#j/iSEViZ]^hZiiiiii*[2DEF a=j) iDAq '1 Qiq5  'q>  )O %Q-($$&$*,  !"#$)% F("!}H   $x:a=[bcdbeYf"g!h h - i0mmV, z"~. "    !oo    t$r;c|r tj|j nd}tt t ||y)Nr)r)r)r1logrrrr:)rBrrs r9rzSequencer2d.init_weightss.37TXXd..//R GMY?Mr;c6td|r ddgSgdS)Nz^stem)z^stages\.(\d+)Nz^norm)i))z^stages\.(\d+)\.blocks\.(\d+)N)z^stages\.(\d+)\.downsample)rr)rr)r)rBcoarses r9 group_matcherzSequencer2d.group_matchers3*$    r;c|rJdy)Nz$gradient checkpointing not supportedrI)rBenables r9set_grad_checkpointingz"Sequencer2d.set_grad_checkpointingsAAAz6r;rGc|jSr?r)rBs r9get_classifierzSequencer2d.get_classifiers yyr;rrcL||_|jj||y)N)r)rrreset)rBrrs r9reset_classifierzSequencer2d.reset_classifiers&  {;r;cl|j|}|j|}|j|}|Sr?)rrrrJs r9forward_featureszSequencer2d.forward_featuress. IIaL KKN IIaLr; pre_logitscN|r|j|dS|j|S)NT)rr)rBrFrs r9 forward_headzSequencer2d.forward_heads$0:tyyty,L ! Lr;cJ|j|}|j|}|Sr?)rrrJs r9rKzSequencer2d.forwards'  ! !! $   a r;F)Tr?)rLrMrNrr{r rrr*rrArrOjitignorerrModulerrxrstrrrryrrKrQrRs@r9rr7s$+)+(r||6gg/R%hN YY     YYBB YY  timm zstages.0.blocks.0.norm1.weightmodelrNzblocks.([0-9]+).([0-9]+).downcDdt|jddzdS)Nrrz.downsample.down)rxgroup)rFs r9rz&checkpoint_filter_fn..s$QWWUVZ[\I\H]]m?nr;zblocks.([0-9]+).([0-9]+)zstages.\1.blocks.\2zhead.zhead.fc.)reitemssubreplace) state_dictrr out_dictkrts r9checkpoint_filter_fnrs':5*(  H  "1 FF35npq r FF.0F J IIgz *  Or;c ttd}|jd|}tt||ft t d|d|}|S)Nrj out_indicesT)flatten_sequentialr)pretrained_filter_fn feature_cfg)rrpoprrrr)variant pretrainedrDdefault_out_indicesrrs r9_create_sequencer2drsYa/**],?@K  2DkJ     E Lr;c :|dddtddttddd |S)Nr)rjrrbicubicTz stem.projzhead.fc) urlrrU pool_sizecrop_pct interpolationfixed_input_sizemeanr first_conv classifier)r rr)rrDs r9_cfgr's3=t$yVZ%.B!     r;ztimm/) hf_hub_id)zsequencer2d_s.in1kzsequencer2d_m.in1kzsequencer2d_l.in1krGc xtgdgdgdgdgdtddd }td d |it|fi|}|S) NrrrYrrrrrTrZ rrrrrrrWr`rcr) sequencer2d_srr{rrrD model_argsrs r9r,r,sK '%' J eJ e$zJd]cJd eE Lr;c ztd gdgdgdgdgdtdddd |}td d |it|fi|}|S) N)rrjrjr*rrrTrZr+rrI) sequencer2d_mr-r.s r9r2r2Z  '%'   J eJ e$zJd]cJd eE Lr;c ztd gdgdgdgdgdtdddd |}td d |it|fi|}|S) N)rrrr*rrrTrZr+rrI) sequencer2d_lr-r.s r9r6r6r3r;)rFr)r)1__doc__r1 functoolsr itertoolsrtypingrrrOtorch.nnr timm.datarrr timm.layersr r r r r_builderr _manipulater _registryrr__all__rrfloatr:r=rTr{rrrrrrrr' default_cfgsr,r2r6rIr;r9rDsg  " SSPP*$< /"))35B"))[ [|YB#ryy#L bii  299 3ryy3l"))D$ %111&            r;