kh dZddlZddlmZddlmZmZmZmZm Z m Z ddl Z ddl m Z ddlmZmZddlmZmZmZmZmZmZmZddlmZdd lmZdd lmZmZm Z dd l!m"Z"m#Z#m$Z$d d gZ%Gdd e jLZ'Gdde jLZ(Gdde jLZ)Gdde jLZ*Gdde jLZ+Gdd e jLZ,dide jLde-de.de/ddf dZ0dZ1djde,fdZ2dkdee-effd Z3e"id!e3d"e3d#e3d$e3d%d&'d(e3d%d)d*+d,e3d-e3d%d.'d/e3d%d0d*+d1e3d%d2d3d4d5d6d78d9e3d%d:d3d4d5d6;d<e3ee=d>e3d%d?ee@dAe3d%dBee@dCe3d%dDee@dEe3d%dFee@dGe3d%dHee@dIe3d%dJee@e3d%dKee@e3d%dLee@e3d%dMee@e3d%dNee@e3d%dOee@e3d%dPee@e3e3d%dQ'e3dR Z4e#djde,fdSZ5e#djde,fdTZ6e#djde,fdUZ7e#djde,fdVZ8e#djde,fdWZ9e#djde,fdXZ:e#djde,fdYZ;e#djde,fdZZe#djde,fd]Z?e#djde,fd^Z@e#djde,fd_ZAe#djde,fd`ZBe#djde,fdaZCe$eDd$d-d9d1dIdbdcdddedfdgdh y)luj MLP-Mixer, ResMLP, and gMLP in PyTorch This impl originally based on MLP-Mixer paper. Official JAX impl: https://github.com/google-research/vision_transformer/blob/linen/vit_jax/models_mixer.py Paper: 'MLP-Mixer: An all-MLP Architecture for Vision' - https://arxiv.org/abs/2105.01601 @article{tolstikhin2021, title={MLP-Mixer: An all-MLP Architecture for Vision}, author={Tolstikhin, Ilya and Houlsby, Neil and Kolesnikov, Alexander and Beyer, Lucas and Zhai, Xiaohua and Unterthiner, Thomas and Yung, Jessica and Keysers, Daniel and Uszkoreit, Jakob and Lucic, Mario and Dosovitskiy, Alexey}, journal={arXiv preprint arXiv:2105.01601}, year={2021} } Also supporting ResMlp, and a preliminary (not verified) implementations of gMLP Code: https://github.com/facebookresearch/deit Paper: `ResMLP: Feedforward networks for image classification...` - https://arxiv.org/abs/2105.03404 @misc{touvron2021resmlp, title={ResMLP: Feedforward networks for image classification with data-efficient training}, author={Hugo Touvron and Piotr Bojanowski and Mathilde Caron and Matthieu Cord and Alaaeldin El-Nouby and Edouard Grave and Armand Joulin and Gabriel Synnaeve and Jakob Verbeek and Hervé Jégou}, year={2021}, eprint={2105.03404}, } Paper: `Pay Attention to MLPs` - https://arxiv.org/abs/2105.08050 @misc{liu2021pay, title={Pay Attention to MLPs}, author={Hanxiao Liu and Zihang Dai and David R. So and Quoc V. Le}, year={2021}, eprint={2105.08050}, } A thank you to paper authors for releasing code and weights. Hacked together by / Copyright 2021 Ross Wightman N)partial)AnyDictListOptionalUnionTuple)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) PatchEmbedMlpGluMlpGatedMlpDropPath lecun_normal_ to_2tuple)build_model_with_cfg)feature_take_indices) named_apply checkpointcheckpoint_seq)generate_default_cfgsregister_modelregister_model_deprecations MixerBlockMlpMixerceZdZdZdeeejdejddfde de de e e e e ffd e d e d e d e d e ddffd Zdej dej fdZxZS)rzResidual Block w/ token mixing and channel MLPs. Based on: 'MLP-Mixer: An all-MLP Architecture for Vision' - https://arxiv.org/abs/2105.01601 ?@ư>epsdimseq_len mlp_ratio mlp_layer norm_layer act_layerdrop drop_pathreturnNc Ft |t|D cgc]} t| |zc} \} } |||_||| |||_|dkDr t |ntj|_ |||_ ||| |||_ ycc} w)aInitialize MixerBlock. Args: dim: Dimension of input features. seq_len: Sequence length. mlp_ratio: Expansion ratios for token mixing and channel MLPs. mlp_layer: MLP layer class. norm_layer: Normalization layer. act_layer: Activation layer. drop: Dropout rate. drop_path: Drop path rate. r+r,r%N) super__init__rintnorm1 mlp_tokensrnnIdentityr-norm2 mlp_channels) selfr&r'r(r)r*r+r,r-x tokens_dim channels_dim __class__s Q/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/models/mlp_mixer.pyr2zMixerBlock.__init__?s. :CI:N#OQCCL#O L_ #GZ9SWX09B),BKKM_ %c<9SWX $PsBr;c ||j|j|j|jddjddz}||j|j |j |z}|S Forward pass.r)r-r5r4 transposer9r8r:r;s r?forwardzMixerBlock.forward^sm ttzz!}/F/Fq!/LMWWXY[\]^ ^ t00A?@ @)__name__ __module__ __qualname____doc__r rr6 LayerNormGELUr3rfloatr typer2torchTensorrF __classcell__r>s@r?rr:s gg!YYYUE%,$778 Y  Y  YYYY Y>%,,rGcdeZdZdZdeddffd ZdejdejfdZxZ S)AffinezAffine transformation layer.r&r.Nct|tjt j dd|f|_tjt jdd|f|_y)zXInitialize Affine layer. Args: dim: Dimension of features. rN) r1r2r6 ParameterrPonesalphazerosbeta)r:r&r>s r?r2zAffine.__init__hsN \\%**aC["9: LLaC[!9: rGr;cXtj|j|j|S)zApply affine transformation.)rPaddcmulr[rYrEs r?rFzAffine.forwardrs}}TYY A66rG) rHrIrJrKr3r2rPrQrFrRrSs@r?rUrUes4&;C;D;77%,,7rGrUceZdZdZdeeejdddfdedede de d e d e d e d e d e ddffd Z de jde jfdZxZS)ResBlockzResidual MLP block w/ LayerScale and Affine 'norm'. Based on: `ResMLP: Feedforward networks for image classification...` - https://arxiv.org/abs/2105.03404 g-C6?r%r&r'r(r)r*r+ init_valuesr,r-r.Nc t |t||z} |||_t j |||_| dkDr t| nt j|_ |||_ ||| |||_ t j|tj|z|_t j|tj|z|_y)aInitialize ResBlock. Args: dim: Dimension of input features. seq_len: Sequence length. mlp_ratio: Channel MLP expansion ratio. mlp_layer: MLP layer class. norm_layer: Normalization layer. act_layer: Activation layer. init_values: Initial values for layer scale. drop: Dropout rate. drop_path: Drop path rate. r%r0N)r1r2r3r4r6Linear linear_tokensrr7r-r8r9rWrPrXls1ls2) r:r&r'r(r)r*r+rar,r- channel_dimr>s r?r2zResBlock.__init__|s2 # /* _ YYw809B),BKKM_ %c;)RVW<< ejjo =><< ejjo =>rGr;c B||j|j|j|j|j ddj ddzz}||j|j |j |j|zz}|SrA)r-rerdr4rDrfr9r8rEs r?rFzResBlock.forwards txx$*<*s r?r2zSpatialGatingUnit.__init__s7 !8x( IIgw/ rGctjj|jjdtjj |jj y)z'Initialize weights for projection gate.r"stdN)r6initnormal_rmweightones_biasr:s r? init_weightszSpatialGatingUnit.init_weightss:  ((d3  diinn%rGr;c|jdd\}}|j|}|j|jdd}||jddzS)zApply spatial gating.rCr&)chunkrlrmrD)r:r;uvs r?rFzSpatialGatingUnit.forwardsTwwqbw!1 IIaL IIakk"b) *1;;r2&&&rG)r.N)rHrIrJrKr6rLr3rOr2rxrPrQrFrRrSs@r?rjrjsUCE,, 0C 0# 04 0SW 0& ''%,,'rGrjceZdZdZdeeejdejddfde de de d e d e d e d e d e ddffd Z de jde jfdZxZS)SpatialGatingBlockzpResidual Block w/ Spatial Gating. Based on: `Pay Attention to MLPs` - https://arxiv.org/abs/2105.08050 r`r"r#r%r&r'r(r)r*r+r,r-r.Nc t |t||z} |||_t t |} ||| || ||_|dkDrt||_ ytj|_ y)aInitialize SpatialGatingBlock. Args: dim: Dimension of input features. seq_len: Sequence length. mlp_ratio: Channel MLP expansion ratio. mlp_layer: MLP layer class. norm_layer: Normalization layer. act_layer: Activation layer. drop: Dropout rate. drop_path: Drop path rate. )r')r+ gate_layerr,r%N) r1r2r3rlrrjr9rr6r7r-) r:r&r'r(r)r*r+r,r-rgsgur>s r?r2zSpatialGatingBlock.__init__sk. # /* sO '9%c;)X[bfg09B),BKKMrGr;cj||j|j|j|z}|SrB)r-r9rlrEs r?rFzSpatialGatingBlock.forwards- t001>? ?rG)rHrIrJrKrrr6rLrMr3rNrOr2rPrQrFrRrSs@r?rrs !&&r||> gg!RRR R  R  RRRR R<%,,rGrc%eZdZdZdddddddeeeejd ejd d d d d d fde de de de de de de e e e e ffdedededede de de dedededd f$fd! Zej&j(d7dedd fd"Zej&j(d7d#edeeeffd$Zej&j(d8d%edd fd&Zej&j(dej4fd'Zd9de deedd fd(Z d:d)ej<d*ee e ee fd+ed,ed-ed.ede eej<e ej<eej<fffd/Z d;d*e e ee fd0ed1edee fd2Z!d)ej<dej<fd3Z"d7d)ej<d4edej<fd5Z#d)ej<dej<fd6Z$xZ%S)s r?r2zMlpMixer.__init__snR &&ENNND1DN"'!%.zD   /6dii.NDII((*T^ mm:& &(  %%#%##(  &( ) Y^^hXikSTD'! yI Nky) I.>IAoBIIi)9)9:SUS^S^S`  t$% &(ks !+E:"E?c|r tj|j nd}tt t ||y)zrInitialize model weights. Args: nlhb: Use negative log bias initialization for head. r%) head_bias)rN)mathlogrrr _init_weights)r:rrs r?rxzMlpMixer.init_weightsAs048TXXd..//R GMY?MrGcoarsec tdddgS)zCreate regex patterns for parameter grouping. Args: coarse: Use coarse grouping. Returns: Dictionary mapping group names to regex patterns. z^stem)z^blocks\.(\d+)N)z^norm)i)rr)r)r:rs r? group_matcherzMlpMixer.group_matcherKs-/CD  rGenablec||_y)zEnable or disable gradient checkpointing. Args: enable: Whether to enable gradient checkpointing. N)r)r:rs r?set_grad_checkpointingzMlpMixer.set_grad_checkpointingZs #)rGc|jS)zGet the classifier module.)rrws r?get_classifierzMlpMixer.get_classifiercsyyrGc||_| |dvsJ||_|dkDr&tj|j||_ytj |_y)zReset the classifier head. Args: num_classes: Number of classes for new classifier. global_pool: Global pooling type. N)rr)rrr6rcrr7r)r:rrs r?reset_classifierzMlpMixer.reset_classifierhsT'  "+- --*D >IAoBIIdnnk: SUS^S^S` rGr;indicesrl stop_early output_fmtintermediates_onlyc $|dvsJd|dk(}g}tt|j|\} } |j\} } } }|j |}t j js|s |j}n|jd| dz}t|D]m\}}|jr+t j js t||}n||}|| vsJ|j|r|j|n|o|ra|jj| |f\}}|Dcgc]6}|j| ||djdddd j!8}}|r|S|j|}||fScc}w) aVForward features that returns intermediates. Args: x: Input image tensor. indices: Take last n blocks if int, all if None, select matching indices if sequence. norm: Apply norm layer to all intermediates. stop_early: Stop iterating over blocks when last desired intermediate hit. output_fmt: Shape of intermediate feature outputs ('NCHW' or 'NLC'). intermediates_only: Only return intermediate features. Returns: List of intermediate features or tuple of (final features, intermediates). )NCHWNLCz)Output format must be one of NCHW or NLC.rNrrzrrrC)rlenrshaperrPjit is_scripting enumeraterrappendrldynamic_feat_sizereshapepermute contiguous)r:r;rrlrrrr intermediates take_indices max_indexBrheightwidthrrblkHWys r?forward_intermediateszMlpMixer.forward_intermediatesus,_,Y.YY,& "6s4;;7G"Q i gg1fe IIaL 99 ! ! #:[[F[[)a-0F' BFAs&&uyy/E/E/GsA&FL $$TTYYq\qA B 99..?DAq^klYZQYYq!Q3;;Aq!QGRRTlMl  IIaL-ms7;F  prune_norm prune_headctt|j|\}}|jd|dz|_|rtj|_|r|j dd|S)aEPrune layers not required for specified intermediates. Args: indices: Indices of intermediate layers to keep. prune_norm: Whether to prune normalization layer. prune_head: Whether to prune the classifier head. Returns: List of indices that were kept. Nrrr)rrrr6r7rlr)r:rrrrrs r?prune_intermediate_layersz"MlpMixer.prune_intermediate_layerss] #7s4;;7G"Q ikk.9q=1  DI   ! !!R (rGc|j|}|jr5tjj st |j |}n|j |}|j|}|S)z/Forward pass through feature extraction layers.)rrrPrrrrrlrEs r?forward_featureszMlpMixer.forward_featuressV IIaL  " "599+A+A+Ct{{A.A AA IIaLrG pre_logitsc|jdk(r|jd}|j|}|r|S|j|S)zForward pass through classifier head. Args: x: Feature tensor. pre_logits: Return features before final classifier. Returns: Output tensor. rrr{)rmeanrr)r:r;rs r? forward_headzMlpMixer.forward_headsD   u $1 A NN1 q0DIIaL0rGcJ|j|}|j|}|Sr)rrrEs r?rFzMlpMixer.forwards'  ! !! $   a rGF)T)N)NFFrF)rFT)&rHrIrJrKrr rr6rLrMr3rrNr rOboolstrr2rPrignorerxrrrrModulerrrrQrrrrrrFrRrSs@r?rrs@ $  ;E *!&r||> gg!$&$&#$%J%J%J% J%  J%  J%J%UE%,$778J%J%J%J%J%J%"J%"J% !J%"#J%$%J%& 'J%X YYNN$NN YY  D  T#s(^     YY)T)T)) YY  aC ahsm aW[ a 8<$$',7 ||7 eCcN347  7  7  7 !%7  tELL!5tELL7I)I#JJ K7 v./$# 3S >*  c 0%,,5<< 1ell 1 1 1%,,rGrnamerflaxr.ct|tjrC|jdrTtjj |j tjj|j|y|rKt|j |jtjj |jytjj|j |jZd|vr,tjj|jdytjj |jyyyt|tjrLt|j |j*tjj |jyyt|tjtjtjfrStjj!|j tjj |jyt#|dr|j%yy)zMixer weight initialization (trying to match Flax defaults). Args: module: Module to initialize. name: Module name. head_bias: Bias value for head layer. flax: Use Flax-style initialization. rNmlpr"rprx) isinstancer6rc startswithrrzeros_rt constant_rvrxavier_uniform_rsConv2drL BatchNorm2d GroupNormrurrx)rrrrs r?rrsw&"))$ ??6 " GGNN6== ) GG  fkk9 5fmm,;;*GGNN6;;/'' 6;;*} >v{{3 + + FBII &fmm$ ;; " GGNN6;; ' # FR\\2>>2<<H I  fmm$ v{{#  (  )rGcJd|vri}|jD]\}}|jdd}|jdd}|jdd}|jdd }|jd s|jd r|jd d d }|||<|S|S)z Remap checkpoints if needed zpatch_embed.proj.weightz patch_embed.zstem.zattn.zlinear_tokens.zmlp.z mlp_channels.gamma_lsz.alphaz.betarrz)itemsreplaceendswithr) state_dictmodelout_dictkrs r?checkpoint_filter_fnrs J.$$& DAq .'2A '#34A &/2A (D)Azz(#qzz'':IIaB'HQK  rGc r|jdd}tt||ftt |dd|}|S)N out_indicesrgetter)r feature_cls)pretrained_filter_fn feature_cfg)poprrrr)variant pretrainedkwargsrrs r? _create_mixerrsJ**]A.K  2[hG     E LrGc "|ddddddddddd |S) Nr)rrr?bicubicT)r r r z stem.projr) urlr input_size pool_sizecrop_pct interpolationfixed_input_sizerrq first_conv classifier)rr s r?_cfgr,s1=tI4!     rGzmixer_s32_224.untrainedzmixer_s16_224.untrainedzmixer_b32_224.untrainedz mixer_b16_224.goog_in21k_ft_in1kztimm/zlhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-vitjx/jx_mixer_b16_224-76587d61.pth) hf_hub_idrzmixer_b16_224.goog_in21kzrhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-vitjx/jx_mixer_b16_224_in21k-617b3de2.pthiSU)rrrzmixer_l32_224.untrainedz mixer_l16_224.goog_in21k_ft_in1kzlhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-vitjx/jx_mixer_l16_224-92f9adc4.pthzmixer_l16_224.goog_in21kzrhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-vitjx/jx_mixer_l16_224_in21k-846aa33c.pthzmixer_b16_224.miil_in21kzvhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-tresnet/mixer_b16_224_miil_in21k-2a558a71.pth)r%r%r%)?rrrbilineari+)rrrrqrrrz mixer_b16_224.miil_in21k_ft_in1kzphttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-tresnet/mixer_b16_224_miil-9229a591.pth)rrrrqrrzgmixer_12_224.untrained)rrqzgmixer_24_224.ra3_in1kzohttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/gmixer_24_224_raa-7daf7ae6.pth)rrrrqzresmlp_12_224.fb_in1kz9https://dl.fbaipublicfiles.com/deit/resmlp_12_no_dist.pthzresmlp_24_224.fb_in1kz9https://dl.fbaipublicfiles.com/deit/resmlp_24_no_dist.pthzresmlp_36_224.fb_in1kz9https://dl.fbaipublicfiles.com/deit/resmlp_36_no_dist.pthzresmlp_big_24_224.fb_in1kz:https://dl.fbaipublicfiles.com/deit/resmlpB_24_no_dist.pthzresmlp_12_224.fb_distilled_in1kz6https://dl.fbaipublicfiles.com/deit/resmlp_12_dist.pthz6https://dl.fbaipublicfiles.com/deit/resmlp_24_dist.pthz6https://dl.fbaipublicfiles.com/deit/resmlp_36_dist.pthz7https://dl.fbaipublicfiles.com/deit/resmlpB_24_dist.pthz6https://dl.fbaipublicfiles.com/deit/resmlpB_24_22k.pthz6https://dl.fbaipublicfiles.com/deit/resmlp_12_dino.pthz6https://dl.fbaipublicfiles.com/deit/resmlp_24_dino.pthznhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/gmlp_s16_224_raa-10536d42.pth) resmlp_24_224.fb_distilled_in1kresmlp_36_224.fb_distilled_in1k#resmlp_big_24_224.fb_distilled_in1k"resmlp_big_24_224.fb_in22k_ft_in1kzresmlp_12_224.fb_dinozresmlp_24_224.fb_dinozgmlp_ti16_224.untrainedzgmlp_s16_224.ra3_in1kzgmlp_b16_224.untrainedc >tddddd|}tdd|i|}|S)zv Mixer-S/32 224x224 Paper: 'MLP-Mixer: An all-MLP Architecture for Vision' - https://arxiv.org/abs/2105.01601 rrrrrr r) mixer_s32_224rrr r  model_argsrs r?r&r&1 KSKFKJ  Oj OJ OE LrGc >tddddd|}tdd|i|}|S)zw Mixer-S/16 224x224 Paper: 'MLP-Mixer: An all-MLP Architecture for Vision' - https://arxiv.org/abs/2105.01601 rrrr%r r) mixer_s16_224r'r(s r?r,r,r*rGc >tddddd|}tdd|i|}|S)zw Mixer-B/32 224x224 Paper: 'MLP-Mixer: An all-MLP Architecture for Vision' - https://arxiv.org/abs/2105.01601 r$ r%r r) mixer_b32_224r'r(s r?r0r01 LcLVLJ  Oj OJ OE LrGc >tddddd|}tdd|i|}|S)z Mixer-B/16 224x224. ImageNet-1k pretrained weights. Paper: 'MLP-Mixer: An all-MLP Architecture for Vision' - https://arxiv.org/abs/2105.01601 rr.r/r%r r) mixer_b16_224r'r(s r?r3r3r1rGc >tddddd|}tdd|i|}|S)zx Mixer-L/32 224x224. Paper: 'MLP-Mixer: An all-MLP Architecture for Vision' - https://arxiv.org/abs/2105.01601 r$r%r r) mixer_l32_224r'r(s r?r7r71 MdMfMJ  Oj OJ OE LrGc >tddddd|}tdd|i|}|S)z Mixer-L/16 224x224. ImageNet-1k pretrained weights. Paper: 'MLP-Mixer: An all-MLP Architecture for Vision' - https://arxiv.org/abs/2105.01601 rr5r6r%r r) mixer_l16_224r'r(s r?r:r:r8rGc htdddddttjd|}t dd|i|}|S) zV Glu-Mixer-12 224x224 Experiment by Ross Wightman, adding SwiGLU to MLP-Mixer rr.rr!rrrr(r)r+r r) gmixer_12_224rrr6SiLUrr(s r?r?r?F 7"zBGG7/57J  Oj OJ OE LrGc htdddddttjd|}t dd|i|}|S) zV Glu-Mixer-24 224x224 Experiment by Ross Wightman, adding SwiGLU to MLP-Mixer rr5r<r=r>r r) gmixer_24_224r@r(s r?rDrDrBrGc Ttdddddttd|}tdd|i|}|S) zx ResMLP-12 Paper: `ResMLP: Feedforward networks for image classification...` - https://arxiv.org/abs/2105.03404 rr.r<r`rrrr(rr*r r) resmlp_12_224)rr_rUrr(s r?rGrGsC u"qhciumsuJ  Oj OJ OE LrGc jtd ddddttdtd|}t d d|i|}|S) zx ResMLP-24 Paper: `ResMLP: Feedforward networks for image classification...` - https://arxiv.org/abs/2105.03404 rr5r<r`gh㈵>rarFr r) resmlp_24_224rrr_rUrr(s r?rJrJL V"qH$7FVNTVJ  Oj OJ OE LrGc jtd ddddttdtd|}t d d|i|}|S) zx ResMLP-36 Paper: `ResMLP: Feedforward networks for image classification...` - https://arxiv.org/abs/2105.03404 r$r<r`r"rIrFr r) resmlp_36_224rKr(s r?rOrOrLrGc jtd ddddttdtd|}t d d|i|}|S) zz ResMLP-B-24 Paper: `ResMLP: Feedforward networks for image classification...` - https://arxiv.org/abs/2105.03404 rr5r/r`r"rIrFr r)resmlp_big_24_224rKr(s r?rQrQsL VsaH$7FVNTVJ  S* S SE LrGc Ttdddddttd|}tdd|i|}|S) zU gMLP-Tiny Paper: `Pay Attention to MLPs` - https://arxiv.org/abs/2105.08050 rrrrr(rr)r r) gmlp_ti16_224rrrrr(s r?rWrWsC &"qN`&$&J  Oj OJ OE LrGc Ttdddddttd|}tdd|i|}|S) zV gMLP-Small Paper: `Pay Attention to MLPs` - https://arxiv.org/abs/2105.08050 rrSrUrVr r) gmlp_s16_224rXr(s r?r[r[*C &"qN`&$&J  NZ N: NE LrGc Ttdddddttd|}tdd|i|}|S) zU gMLP-Base Paper: `Pay Attention to MLPs` - https://arxiv.org/abs/2105.08050 rrSrrUrVr r) gmlp_b16_224rXr(s r?r^r^6r\rGrr r!r"rGrJ) mixer_b16_224_in21kmixer_l16_224_in21kmixer_b16_224_miilmixer_b16_224_miil_in21kresmlp_12_distilled_224resmlp_24_distilled_224resmlp_36_distilled_224resmlp_big_24_distilled_224resmlp_big_24_224_in22ft1kresmlp_12_224_dinoresmlp_24_224_dino)r%Fr)r)ErKr functoolsrtypingrrrrrr rPtorch.nnr6 timm.datar r timm.layersr r rrrrr_builderr _featuresr _manipulaterrr _registryrrr__all__rrrUr_rjrrrrNrrrrr default_cfgsr&r,r0r3r7r:r?rDrGrJrOrQrWr[r^rHrrGr?rusH'P :: A]]]*+@@YY  $((V7RYY7$,ryy,^' '@&&Rtryytn%"))%3%5%T%^b%P" ( d38n%a&tva&tva&tva&' z) a& A!a&tva&' z)a&$ A!%a&2 E |e:ch!3a&<' ~ |e:)=a&Ht)>DXYIa&Jd } "(<>Ka&TT G "(<>Ua&\T G "(< >]a&fT G "(<>ga&n  H "(<">oa&x&t D "(<(>ya&@(, D "(<(>(, D "(<(>,0 E "(<,> +/ D "(<+> " D "(<>" D "(<> $v! |#fAa&a HXH==< :@@@#H"F)) ' rG