khPdZddlZddlmZmZmZmZmZmZm Z ddl Z ddl m Z ddl m cmZddlmZmZddlmZmZmZmZmZmZddlmZddlmZdd lmZdd l m!Z!m"Z"d gZ#Gd d e jHZ%Gdde jHZ&Gdde jHZ'Gdde jHZ(Gdde jHZ)Gdde jHZ*de+dede jHfdZ,dZdee-dfde.de-dee-e-e-e-ffd Z/Gd!d"e jHZ0Gd#d$e jHZ1 d[d%ed&e-d'e-d(ee-d)e-d*e-d+e-d,e-d-e.d.e2d/e.d0e.d1ede jffd2Z4 d\d%ed&e-d'e-d(ee-d)e-d-e.d.e2d/e.d0e.d1ede jffd3Z5Gd4d e jHZ6d]d5e+d6e2d1ede6fd7Z7d^d8e+d1edee+effd9Z8e"e8d:d;d<=e8d:d>d?d@Ae8d:dBd<=e8d:dCd?d@Ae8d:dDd<=e8d:dEd?dFAe8d:dGd<=e8d:dHdIdFAe8d:dJd<=e8d:dKdIdFAe8d:dLdIdMAdN Z9e!d]d6e2d1ede6fdOZ:e!d]d6e2d1ede6fdPZ;e!d]d6e2d1ede6fdQZe!d]d6e2d1ede6fdTZ?e!d]d6e2d1ede6fdUZ@e!d]d6e2d1ede6fdVZAe!d]d6e2d1ede6fdWZBe!d]d6e2d1ede6fdXZCe!d]d6e2d1ede6fdYZDy)_a5 Vision OutLOoker (VOLO) implementation Paper: `VOLO: Vision Outlooker for Visual Recognition` - https://arxiv.org/abs/2106.13112 Code adapted from official impl at https://github.com/sail-sg/volo, original copyright in comment below Modifications and additions for timm by / Copyright 2022, Ross Wightman N)AnyCallableDictListOptionalTupleUnionIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)DropPathMlp to_2tuple to_ntuple trunc_normal_use_fused_attn)build_model_with_cfg)feature_take_indices) checkpoint)register_modelgenerate_default_cfgsVOLOceZdZdZ ddededededededed effd Zd ejd ejfd Z xZ S)OutlookAttentionz,Outlook attention mechanism for VOLO models.dim num_heads kernel_sizepaddingstrideqkv_bias attn_drop proj_dropc t |||z} ||_||_||_||_| dz|_tj||||_ tj||dz|z|_ tj||_ tj|||_ tj||_tj||||_tj"||d|_y)aInitialize OutlookAttention. Args: dim: Input feature dimension. num_heads: Number of attention heads. kernel_size: Kernel size for attention computation. padding: Padding for attention computation. stride: Stride for attention computation. qkv_bias: Whether to use bias in linear layers. attn_drop: Attention dropout rate. proj_drop: Projection dropout rate. ࿩bias)rrr T)rr ceil_modeN)super__init__rrrr scalennLinearvattnDropoutr"projr#Unfoldunfold AvgPool2dpool) selfrrrrr r!r"r#head_dim __class__s L/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/models/volo.pyr+zOutlookAttention.__init__*s. )#"&  % 3(3IIc;!#3i#?@ I.IIc3' I.iiKQWX LLVFdS xreturnc|j\}}}}|j|jdddd}tj||j z tj||j z }}|j |j||j||jz|j|jz||zjddddd}|j|jddddjdddd} |j| j|||z|j|j|jz|j|jzjddddd} | |jz} | jd} |j| } | |zjdddddj|||jz|jz||z}tj |||f|j|j"|j }|j%|jdddd}|j'|}|S) Forward pass. Args: x: Input tensor of shape (B, H, W, C). Returns: Output tensor of shape (B, H, W, C). rrr(r) output_sizerrr )shaper/permutemathceilr r4reshaperrr6r0r,softmaxr"Ffoldrr2r#) r7r<BHWCr/hwr0s r:forwardzOutlookAttention.forwardSs!WW 1a FF1I  aAq )yyT[[)499Q_+E1 KKN " " t~~qDNN2   t// /Q 88?1aA8N yy1aA./771aCyy&& q1udnnd&6&69I9I&I   t// /118Aq!Q1G djj |||#~~d# AX  q!Q1 - 5 5aT=M=M9MPTP`P`9`bcfgbg h FF11a&d6F6FPTP\P\eiepep q IIaii1a+ , NN1 r;)r@rrFrT) __name__ __module__ __qualname____doc__intboolfloatr+torchTensorrS __classcell__r9s@r:rr's6 !"!!'T'T'T 'T  'T  'T'T'T'TR  %,, r;rceZdZdZdddddej ej dfdededed ed ed ed ed ede de de ffd Z de jde jfdZxZS) Outlookerz9Outlooker block that combines outlook attention with MLP.r@rTFrrrr r mlp_ratior" drop_path act_layer norm_layerr!c rt || ||_t|||||| ||_|dkDr t |nt j|_| ||_ t|t||z| |_ |dkDrt ||_ yt j|_ y)afInitialize Outlooker block. Args: dim: Input feature dimension. kernel_size: Kernel size for outlook attention. padding: Padding for outlook attention. stride: Stride for outlook attention. num_heads: Number of attention heads. mlp_ratio: Ratio for MLP hidden dimension. attn_drop: Attention dropout rate. drop_path: Stochastic depth drop rate. act_layer: Activation layer type. norm_layer: Normalization layer type. qkv_bias: Whether to use bias in linear layers. )rrr r!r"rT in_featureshidden_featuresreN)r*r+norm1rr0r r-Identity drop_path1norm2rrYmlp drop_path2) r7rrrr rrcr"rdrerfr!r9s r:r+zOutlooker.__init__ys: _ $  #  2;R(9-R[[]_ i0  2;R(9-R[[]r;r<r=c||j|j|j|z}||j|j |j |z}|SzoForward pass. Args: x: Input tensor. Returns: Output tensor. rmr0rkrprornr7r<s r:rSzOutlooker.forwardQ  $**Q- 89 9 A 78 8r;)rUrVrWrXr-GELU LayerNormrYr[rrZr+r\r]rSr^r_s@r:raravsC!!!"$''#%<<"0S0S0S 0S  0S  0S0S0S0S 0S!0S0Sd  %,, r;rac eZdZUdZej j eed< d de de dede de f fd Z d ejd ejfd Z xZS) Attentionz!Multi-head self-attention module. fused_attnrrr!r"r#cJt|||_||z}|dz|_t |_t j||dz||_t j||_ t j|||_ t j||_ y)a,Initialize Attention module. Args: dim: Input feature dimension. num_heads: Number of attention heads. qkv_bias: Whether to use bias in QKV projection. attn_drop: Attention dropout rate. proj_drop: Projection dropout rate. r%r@r&N) r*r+rr,rrzr-r.qkvr1r"r2r#)r7rrr!r"r#r8r9s r:r+zAttention.__init__s" ")#% (*99S#'9I.IIc3' I.r;r<r=c|j\}}}}|j|j|||zd|j||jzj ddddd}|j d\}}} |j r>$'DqA KK1  % %aAq 1 IIaL NN1 r;)FrTrT)rUrVrWrXr\jitFinalrZ__annotations__rYr[r+r]rSr^r_s@r:ryrysw+ %% "!! /// /  /  /8%,,r;ryceZdZdZddddej ej fdededede d ed ed e d e ffd Z de jde jfdZxZS) Transformerz9Transformer block with multi-head self-attention and MLP.@FrTrrrcr!r"rdrerfc lt ||||_t|||||_|dkDr t |nt j|_|||_ t|t||z||_ |dkDrt ||_ yt j|_ y)aInitialize Transformer block. Args: dim: Input feature dimension. num_heads: Number of attention heads. mlp_ratio: Ratio for MLP hidden dimension. qkv_bias: Whether to use bias in QKV projection. attn_drop: Attention dropout rate. drop_path: Stochastic depth drop rate. act_layer: Activation layer type. norm_layer: Normalization layer type. )rr!r"rTrhN)r*r+rkryr0r r-rlrmrnrrYrorp) r7rrrcr!r"rdrerfr9s r:r+zTransformer.__init__s. _ cYU^_ 1:R(9-R[[]_ 3C)O8LXab1:R(9-R[[]r;r<r=c||j|j|j|z}||j|j |j |z}|Srrrsrts r:rSzTransformer.forwardrur;)rUrVrWrXr-rvrwrYr[rZrr+r\r]rSr^r_s@r:rrsC ""!!"$''#%<<SSS S  S  SS S!S@  %,, r;rc eZdZdZ d dededeedededef fd Zd e jd e jfd Z xZ S) ClassAttentionz6Class attention mechanism for class token interaction.rrr8r!r"r#ct|||_|||_n ||z}||_|dz|_t j ||j|jzdz||_t j ||j|jz||_t j||_ t j |j|jz||_ t j||_ y)a{Initialize ClassAttention. Args: dim: Input feature dimension. num_heads: Number of attention heads. head_dim: Dimension per head. If None, computed as dim // num_heads. qkv_bias: Whether to use bias in QKV projection. attn_drop: Attention dropout rate. proj_drop: Projection dropout rate. Nr%rAr&) r*r+rr8r,r-r.kvrr1r"r2r#)r7rrr8r!r"r#r9s r:r+zClassAttention.__init__.s& "  $DMi'H$DM% ))C!?!!C(S3  >XNI.IIdmmdnn>1dmm Ltzz Y1;;r2&&|||#~~d#AX((A.66q!T]]T^^=[\ IIi( NN9- r;)rNFrTrT) rUrVrWrXrYrrZr[r+r\r]rSr^r_s@r:rr+sv@ &*"!! / / /sm /  /  / /D%,,r;rceZdZdZddddddej ej fdededeed e d e d e d e d e de de ffd Z de jde jfdZxZS) ClassBlockz3Class block that combines class attention with MLP.NrFrTrrr8rcr!dropr"rdrerfc rt || ||_t|||||||_|dkDr t |nt j|_| ||_ t|t||z| ||_ |dkDrt ||_ yt j|_ y)a1Initialize ClassBlock. Args: dim: Input feature dimension. num_heads: Number of attention heads. head_dim: Dimension per head. If None, computed as dim // num_heads. mlp_ratio: Ratio for MLP hidden dimension. qkv_bias: Whether to use bias in QKV projection. drop: Dropout rate. attn_drop: Attention dropout rate. drop_path: Stochastic depth drop rate. act_layer: Activation layer type. norm_layer: Normalization layer type. )rr8r!r"r#rT)rirjrerN)r*r+rkrr0r r-rlrmrnrrYrorp) r7rrr8rcr!rr"rdrerfr9s r:r+zClassBlock.__init__ls6 _ "    2;R(9-R[[]_ i0   2;R(9-R[[]r;r<r=c$|ddddf}||j|j|j|z}||j|j |j |z}t j||ddddfgdS)zForward pass. Args: x: Input tensor of shape (B, N, C) where first token is class token. Returns: Output tensor with updated class token. NrrC)rmr0rkrprornr\cat)r7r<rs r:rSzClassBlock.forwards~a!eH  $**Q-0H II I9N0O PP yy)Qq!"uX.A66r;)rUrVrWrXr-rvrwrYrr[rZrr+r\r]rSr^r_s@r:rris= '+!"!!"$''#%<<.S.S.Ssm .S  .S  .S.S.S.S .S!.S` 7 7%,, 7r;r block_typekargsr=c $|dk(r tdi|Sy)zGet block based on type. Args: block_type: Type of block ('ca' for ClassBlock). **kargs: Additional keyword arguments for block. Returns: The requested block module. caN)r)rrs r: get_blockrsT"E""r;size.lamr,c|d|z}|d|z}tj|tj}tj|tj}tjd|z }||zj }||zj } tj d|d} tj d|d} tj | |dzz d|} tj | | dzz d|} tj | |dzzd|}tj | | dzzd|}| j| j|j|jfS)a1Get random bounding box for token labeling. Reference: https://github.com/zihangJiang/TokenLabeling Args: size: Input tensor size tuple. lam: Lambda parameter for cutmix. scale: Scaling factor. Returns: Bounding box coordinates (bbx1, bby1, bbx2, bby2). rrA)dtype?rr)r\tensorfloat32sqrtrYrandintclampitem)rrr,rOrNW_tH_tcut_ratcut_wcut_hcxcybbx1bby1bbx2bby2s r: rand_bboxrs3 Q5A Q5A ,,q .C ,,q .Cjjc"G 7]   !E 7]   !E q!T "B q!T "B ;;rEQJ1 -D ;;rEQJ1 -D ;;rEQJ1 -D ;;rEQJ1 -D 99; TYY[$))+ ==r;ceZdZdZ d dededededededeffd Zd ejd ejfd Z xZ S) PatchEmbedz6Image to patch embedding with multi-layer convolution.img_size stem_conv stem_stride patch_sizein_chans hidden_dim embed_dimcnt||dvsJ|rtjtj||d|ddtj |tj dtj||ddddtj |tj dtj||ddddtj |tj d |_nd |_tj||||z||z |_||z||zz|_ y ) aInitialize PatchEmbed. Different from ViT which uses 1 conv layer, VOLO uses multiple conv layers for patch embedding. Args: img_size: Input image size. stem_conv: Whether to use stem convolution layers. stem_stride: Stride for stem convolution. patch_size: Patch size (must be 4, 8, or 16). in_chans: Number of input channels. hidden_dim: Hidden dimension for stem convolution. embed_dim: Output embedding dimension. )r(rr@F)rr rr'T)inplacerNrr ) r*r+r- SequentialConv2d BatchNorm2dReLUconvr2 num_patches) r7rrrrrrrr9s r:r+zPatchEmbed.__init__s. Z'''   (JAk[\chiz*% *jaST[`az*% *jaST[`az*% DIDIII  z[/HQ[_jQjl $ 2x:7MNr;r<r=cb|j|j|}|j|}|S)zForward pass. Args: x: Input tensor of shape (B, C, H, W). Returns: Output tensor of shape (B, embed_dim, H', W'). )rr2rts r:rSzPatchEmbed.forward s- 99  ! A IIaLr;)Frrr@@) rUrVrWrXrYrZr+r\r]rSr^r_s@r:rrs@ #   *O*O*O *O  *O  *O*O*OX  %,, r;rcjeZdZdZd dededeffd ZdejdejfdZxZ S) Downsamplez#Downsampling module between stages. in_embed_dim out_embed_dimrc^t|tj|||||_y)zInitialize Downsample. Args: in_embed_dim: Input embedding dimension. out_embed_dim: Output embedding dimension. patch_size: Patch size for downsampling. rN)r*r+r-rr2)r7rrrr9s r:r+zDownsample.__init__s' IIlMzZde r;r<r=cx|jdddd}|j|}|jdddd}|S)zForward pass. Args: x: Input tensor of shape (B, H, W, C). Returns: Output tensor of shape (B, H', W', C'). rr@rrA)rFr2rts r:rSzDownsample.forward&s> IIaAq ! IIaL IIaAq !r;)rA) rUrVrWrXrYr+r\r]rSr^r_s@r:rrs?- fS f f# f  %,, r;rblock_fnindexrlayersrrrr rcr!r"drop_path_ratekwargsc  g} t||D]E}| |t|d|zzt|dz z }| j|||||||| | | Gtj| } | S)aGenerate outlooker layers for stage 1. Args: block_fn: Block function to use (typically Outlooker). index: Index of current stage. dim: Feature dimension. layers: List of layer counts for each stage. num_heads: Number of attention heads. kernel_size: Kernel size for outlook attention. padding: Padding for outlook attention. stride: Stride for outlook attention. mlp_ratio: Ratio for MLP hidden dimension. qkv_bias: Whether to use bias in QKV projection. attn_drop: Attention dropout rate. drop_path_rate: Stochastic depth drop rate. **kwargs: Additional keyword arguments. Returns: Sequential module containing outlooker blocks. Nr)rrr rrcr!r"rdrangesumappendr-r)rrrrrrrr rcr!r"rrblocks block_idx block_dprs r:outlooker_blocksr5sFF6%=)  "i#fVen2E&EF#f+XY/Z  h #    ]]F #F Mr;c g} t||D]B} || t|d|zzt|dz z } | j||||||| Dtj| } | S)aeGenerate transformer layers for stage 2. Args: block_fn: Block function to use (typically Transformer). index: Index of current stage. dim: Feature dimension. layers: List of layer counts for each stage. num_heads: Number of attention heads. mlp_ratio: Ratio for MLP hidden dimension. qkv_bias: Whether to use bias in QKV projection. attn_drop: Attention dropout rate. drop_path_rate: Stochastic depth drop rate. **kwargs: Additional keyword arguments. Returns: Sequential module containing transformer blocks. Nr)rcr!r"rdr) rrrrrrcr!r"rrrrrs r:transformer_blocksrjs:F6%=)  "i#fVen2E&EF#f+XY/Z  h       ]]F #F Mr;c-eZdZdZddddddddd d d d d d d d ej d dd dfdeededededededede eede eede e dfde e dfde de de de d e d!e d"e d#e e edfd$e d%e d&ef,fd' Zd(ejd)dfd*Zej$j&d)efd+Zej$j&dCd,e d)eeeffd-Zej$j&dDd.e d)dfd/Zej$j&d)ejfd0ZdEdede ed)dfd1Zd2ej8d)ej8fd3Zd2ej8d)ej8fd4Zd2ej8d)eej8e ej8ej8e eeeeffffd5Z dFd2ej8d6e eeeefd7e d8e d9ed:e d)eeej8e ej8eej8fffd;Z! dGd6eeeefdZ"d2ej8d)ej8fd?Z#dCd2ej8d@e d)ej8fdAZ$d2ej8d)ej8fdBZ%xZ&S)HrzVision Outlooker (VOLO) model.rr@tokenrrN)TFFFrbFrT)rrTrArrr num_classes global_poolrstem_hidden_dim embed_dimsr downsamples.outlook_attentionrcr! drop_rate pos_drop_rateattn_drop_raterrf post_layers use_aux_head use_mix_token pooling_scalec|t|t|}t|| } t |}||_||_||_||_|dx|_ |_ |rd|_ |dk(sJdd|_ tdd||||d |_|}|d|z|z|d |z|zf}tj t#j$d |d|d |d|_tj(| |_g|_g|_g}d}t1t|D]}| |r"t3t4||||| || || || }n"t7t8||||| || || ||| }|j;||j,j;||j.j;t=|||d||d z }| |s|j;t?||||d zd|dz}|d z }tj@||_!d|_"|tj@t1t|Dcgc]!}tG|||d| d| d| |d|#c}|_"tj t#j$d d |d|_$tK|jHd|r?|dkDr tjL|j|ntjN|_(nd|_(||j|_)tj(||_*|dkDr tjL|j|ntjN|_+tK|j&d|jY|jZycc}w)a/Initialize VOLO model. Args: layers: Number of blocks in each stage. img_size: Input image size. in_chans: Number of input channels. num_classes: Number of classes for classification. global_pool: Global pooling type ('token', 'avg', or ''). patch_size: Patch size for patch embedding. stem_hidden_dim: Hidden dimension for stem convolution. embed_dims: List of embedding dimensions for each stage. num_heads: List of number of attention heads for each stage. downsamples: Whether to downsample between stages. outlook_attention: Whether to use outlook attention in each stage. mlp_ratio: Ratio for MLP hidden dimension. qkv_bias: Whether to use bias in QKV projection. drop_rate: Dropout rate. pos_drop_rate: Position embedding dropout rate. attn_drop_rate: Attention dropout rate. drop_path_rate: Stochastic depth drop rate. norm_layer: Normalization layer type. post_layers: Post-processing layer types. use_aux_head: Whether to use auxiliary head. use_mix_token: Whether to use token mixing for training. pooling_scale: Pooling scale factor. rBrrz)return all tokens if mix_token is enabledFTrAr)rrrrrrr)r)rcr!r"rf)rcr!rr"rfznetwork.)num_chs reductionmoduleNrT)rrrcr!r"rdrf{Gz?std).r*r+lenrrrr mix_tokenr num_featureshead_hidden_sizebetagrad_checkpointingr patch_embedr- Parameterr\zeros pos_embedr1pos_drop stage_ends feature_inforrrarrrdictr ModuleListnetwork post_networkr cls_tokenrr.rlaux_headnorm head_dropheadapply _init_weights)r7rrrrrrrrrrrrcr!rrrrrfrrrr num_layersr patch_gridrristager9s r:r+z VOLO.__init__sf [ )Ij))4 X&&&&*4>rNBD1 DI') V+V V)"'%!& m   qkZ/=@(1+Q[B[_lBlm ekk!Z]JqMS]^`Sa&bc ]3  s6{#$ A #(qMaL'l%,) +qMaL'l%#1,)  NN5 ! OO " "9 -    $ $T*Q-1W_`i_jUk%l m NI1~z*Q-AE9JANOQQ I$ L}}W- !  " " s;/0 /N"2'm'm%, )+ / !D  \\%++aJrN*KLDN $..c 2 ITWXBIId&7&7E^`^i^i^kDM DMt001 I.ALqBIId//=VXVaVaVc dnn#. 4%%&7 /s&N9mr=c t|tjrjt|jdt|tjr8|j +tj j|j dyyyy)z\Initialize weights for modules. Args: m: Module to initialize. rrNr) isinstancer-r.rweightr'init constant_)r7rs r:rzVOLO._init_weights7s\ a # !(( ,!RYY'AFF,>!!!&&!,-?' $r;c ddhS)zwGet set of parameters that should not have weight decay. Returns: Set of parameter names. r rrr7s r:no_weight_decayzVOLO.no_weight_decayBs[))r;coarsec&tdddggdS)zGet parameter grouping for optimizer. Args: coarse: Whether to use coarse grouping. Returns: Parameter grouping dictionary. z ^cls_token|pos_embed|patch_embed)z^network\.(\d+)\.(\d+)N)z^network\.(\d+)r))z ^cls_tokenr()z^post_network\.(\d+)N)z^norm)i)stemrblocks2)r )r7r&s r: group_matcherzVOLO.group_matcherKs$41*  r;enablec||_y)zqSet gradient checkpointing. Args: enable: Whether to enable gradient checkpointing. N)r)r7r,s r:set_grad_checkpointingzVOLO.set_grad_checkpointingbs #)r;c|jS)zYGet classifier module. Returns: The classifier head module. )rr$s r:get_classifierzVOLO.get_classifierksyyr;c6||_|||_|dkDr tj|j|ntj |_|j?|dkDr tj|j|ntj |_yy)zReset classifier head. Args: num_classes: Number of classes for new classifier. global_pool: Global pooling type. Nr)rrr-r.rrlrr)r7rrs r:reset_classifierzVOLO.reset_classifierts~'  "*D ALqBIId//=VXVaVaVc == $ITWXBIId&7&7E^`^i^i^kDM %r;r<cPt|jD]i\}}|dk(r ||jz}|j|}|jr+t j js t||}b||}k|j\}}}}|j|d|}|S)zForward pass through token processing stages. Args: x: Input tensor of shape (B, H, W, C). Returns: Token tensor of shape (B, N, C). rArB) enumeraterr r rr\r is_scriptingrrErI)r7r<idxblockrMrNrOrPs r:forward_tokenszVOLO.forward_tokenss$DLL1 JCax&MM!$&&uyy/E/E/Gua(!H WW 1a IIaQ r;c2|j\}}}|jj|dd}tj||gd}|j D]A}|j r+tjjs t||}:||}C|S)zForward pass through class attention blocks. Args: x: Input token tensor of shape (B, N, C). Returns: Output tensor with class token of shape (B, N+1, C). rBrrC) rErexpandr\rrrrr5r)r7r<rMrrP cls_tokensr7s r: forward_clszVOLO.forward_clss''1a^^**1b"5 IIz1o1 -&& E&&uyy/E/E/Gua(!H   r;c |j|}|jdddd}|jr0|jr#tj j |j|jj}|jd|jz|jd|jz}}t|j||j\}}}}|j} |j|z|j|z} } |j|z|j|z} } |jddd| | | | ddf| dd| | | | ddf<| }nd\}}}}|j|}|j |j#|}|j%|}|j&dk(r|j)d }n|j&d k(r |dddf}n|}|j*|S|j+|ddddf}|js|d |j-ddzzS|jr|jr|j/|jd|jd }|j} |jddd||||ddf| dd||||ddf<| }|j/|jd||z|jd }||||||ffS) a Forward pass for training with mix token support. Args: x: Input tensor of shape (B, C, H, W). Returns: If training with mix_token: tuple of (class_token, aux_tokens, bbox). Otherwise: class_token tensor. rrAr@r)r,N)rrrravgrCr?rB)rrFrrr\ distributionsBetarsamplerErrrcloneflipr8rr<rrmeanrmaxrI)r7r<rpatch_hpatch_wrrrrtemp_xsbbx1sbby1sbbx2sbby2x_clsx_auxs r: forward_trainzVOLO.forward_trains    Q  IIaAq ! >>dmm%%**499dii@GGIC wwqzT-?-??tOaOaAaWG%.qvvxDDVDV%W "D$dWWYF--4d6H6H46O5E--4d6H6H46O5E56VVAYq%+uUZ{\]?]5^F1eEk5;1 2A%/ "D$d    "    (  #A IIaL   u $FFqFME    (adGEE == L a12h'}}31a00 0 >>dmmMM%++a.'7EKKPROTE[[]F16Aq$t)TRVYXY?Y1ZF1d4idA- .EMM%++a.'G2CU[[QS_UEedD$555r;indicesr stop_early output_fmtintermediates_onlyc |dvsJdg}tt|j|\}} |D cgc]} |j| }} |j| } |j\} } } }|j |j dddd}t jjs|s |j}n|jd| dz}t|D]\}}|dk(r ||jz}|j|}|jr+t jjs t||}n||}||vso|r|dk\r|j|}n|}|j!|j dddd|r|S|j\} }}}|j#| d|}|j$|j'|}|j|}||fScc} w) a Forward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if int, all if None, select matching indices if sequence norm: Apply norm layer to all intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features Returns: )NCHWzOutput format must be NCHW.rrAr@rNrB)rrr rErrFr\rr5rr4r r rrrrrIrr<)r7r<rQrrRrSrT intermediates take_indices max_indexrrM_heightwidthrr6r7x_interrNrOrPs r:forward_intermediateszVOLO.forward_intermediatess*Y&E(EE& "6s4??7KW"U i4@Aq*A AOOI.  gg1fe   Q  ' '1a 3 99 ! ! #:llGll>IM2G#G, BJCax&MM!$&&uyy/E/E/Gua(!Hl"C1H"iilGG$$W__Q1a%@A B  WW 1a IIaQ     (  #A IIaL-OBsG" prune_norm prune_headc( tt|j|\}}|j|}|jd|dz|_|rt j |_|r+t j|_|jdd|S)aHPrune layers not required for specified intermediates. Args: indices: Indices of intermediate layers to keep. prune_norm: Whether to prune normalization layer. prune_head: Whether to prune classification head. Returns: List of kept intermediate indices. Nrr) rrr rr-rlrrrr2)r7rQr_r`rXrYs r:prune_intermediate_layerszVOLO.prune_intermediate_layers*s|  "6s4??7KW"U iOOI. ||NY]3  DI  " D   ! !!R (r;c|j|jdddd}|j|}|j|j |}|j |}|S)zForward pass through feature extraction. Args: x: Input tensor of shape (B, C, H, W). Returns: Feature tensor. rrAr@r)rrFr8rr<rrts r:forward_featureszVOLO.forward_featuresFsg   Q  ' '1a 3    "    (  #A IIaLr; pre_logitscL|jdk(r|jd}n|jdk(r |dddf}n|}|j|}|r|S|j|}|j4|j |ddddf}|d|j ddzz}|S)zForward pass through classification head. Args: x: Input feature tensor. pre_logits: Whether to return pre-logits features. Returns: Classification logits or pre-logits features. r>rrCrNrr?)rrErrrrF)r7r<rfoutauxs r: forward_headzVOLO.forward_headZs   u $&&Q&-C    (AqD'CC NN1  Jiin == $--!QR%)Ccggajm++C r;cL |j|}|j|}|S)zForward pass (simplified, without mix token training). Args: x: Input tensor of shape (B, C, H, W). Returns: Classification logits. )rerjrts r:rSz VOLO.forwardts- @  ! !! $   a r;F)T)N)NFFrVF)rFT)'rUrVrWrXr-rwrrYstrrrrZr[rr+Modulerr\rignoresetr%rrr+r.r0r2r]r8r<r rPr^rcrerjrSr^r_s@r:rrs)(  #&#%.2-1,G2M""!#%$&$&#%<<5A!%"'!"/\'I\'\' \'  \'  \'\'!\'!c+\' S *\'tSy)\' %T3Y/\'\'\'\' !!\'""#\'$"%\'&!'\'("%S/2)\'*+\', -\'./\'| -ryy -T - YY*** YY D T#s(^  , YY)T)T)) YY  lC lhsm lW[ l .U\\ell&;6u||;6ellE%,,X]XdXdfkloqtvy{~l~fJEA7A1B;6@8<$$',? ||? eCcN34?  ?  ?  ? !%?  tELL!5tELL7I)I#JJ K? F./$# 3S >*  c 8%,,5<<(ell4  %,, r;variant pretrainedc d|jdd}tt||fdt|di|S)zCreate VOLO model. Args: variant: Model variant name. pretrained: Whether to load pretrained weights. **kwargs: Additional model arguments. Returns: VOLO model instance. out_indicesr@ feature_cfggetter)rt feature_cls)poprrr )rqrrrrts r: _create_volorysE**]A.K   [hG     r;urlc 2|ddddddttddd |S) zCreate model configuration. Args: url: URL for pretrained weights. **kwargs: Additional configuration options. Returns: Model configuration dictionary. r)r@rrNQ?bicubicTzpatch_embed.conv.0)rr) rzr input_size pool_sizecrop_pct interpolationfixed_input_sizerEr first_conv classifierr )rzrs r:_cfgrs5=t)%.B*:N     r;ztimm/zLhttps://github.com/sail-sg/volo/releases/download/volo_1/d1_224_84.2.pth.tarr|) hf_hub_idrzrzLhttps://github.com/sail-sg/volo/releases/download/volo_1/d1_384_85.2.pth.tarr)r@rr)rrzrr~zLhttps://github.com/sail-sg/volo/releases/download/volo_1/d2_224_85.2.pth.tarzLhttps://github.com/sail-sg/volo/releases/download/volo_1/d2_384_86.0.pth.tarzLhttps://github.com/sail-sg/volo/releases/download/volo_1/d3_224_85.4.pth.tarzLhttps://github.com/sail-sg/volo/releases/download/volo_1/d3_448_86.3.pth.tar)r@rzLhttps://github.com/sail-sg/volo/releases/download/volo_1/d4_224_85.7.pth.tarzMhttps://github.com/sail-sg/volo/releases/download/volo_1/d4_448_86.79.pth.targffffff?zMhttps://github.com/sail-sg/volo/releases/download/volo_1/d5_224_86.10.pth.tarzLhttps://github.com/sail-sg/volo/releases/download/volo_1/d5_448_87.0.pth.tarzMhttps://github.com/sail-sg/volo/releases/download/volo_1/d5_512_87.07.pth.tar)r@r) zvolo_d1_224.sail_in1kzvolo_d1_384.sail_in1kzvolo_d2_224.sail_in1kzvolo_d2_384.sail_in1kzvolo_d3_224.sail_in1kzvolo_d3_448.sail_in1kzvolo_d4_224.sail_in1kzvolo_d4_448.sail_in1kzvolo_d5_224.sail_in1kzvolo_d5_448.sail_in1kzvolo_d5_512.sail_in1kc >tddddd|}tdd|i|}|S)VOLO-D1 model, Params: 27M.r(r(rrArrr rrrrrrrr) volo_d1_224r ryrrr model_argsmodels r:rr4p\6JVepiopJ  L: L LE Lr;c >tddddd|}tdd|i|}|S)rrrrrrrr) volo_d1_384rrs r:rrrr;c >tddddd|}tdd|i|}|S)VOLO-D2 model, Params: 59M.rr( r(rrrrrrrrrrr) volo_d2_224rrs r:rr4q]7KWfqjpqJ  L: L LE Lr;c >tddddd|}tdd|i|}|S)rrrrrrrr) volo_d2_384rrs r:rrrr;c >tddddd|}tdd|i|}|S)VOLO-D3 model, Params: 86M.rrrr(rrrrrr) volo_d3_224rrs r:rrrr;c >tddddd|}tdd|i|}|S)rrrrrrrr) volo_d3_448rrs r:rrrr;c >tddddd|}tdd|i|}|S)VOLO-D4 model, Params: 193M.rrrrrrrrrrrr) volo_d4_224rrs r:rr 4r]7KWgrkqrJ  L: L LE Lr;c >tddddd|}tdd|i|}|S)rrrrrrrr) volo_d4_448rrs r:rrrr;c Btddddddd|}td d|i|}|S) jVOLO-D5 model, Params: 296M. stem_hidden_dim=128, the dim in patch embedding is 128 for VOLO-D5. rrr(rrr(rrrrcrrrr) volo_d5_224rrs r:rrB 4+?K[S4,24J  L: L LE Lr;c Btddddddd|}td d|i|}|S) rrrrr(rrrrr) volo_d5_448rrs r:rr)rr;c Btddddddd|}td d|i|}|S) rrrrr(rrrrr) volo_d5_512rrs r:rr6rr;r)rr@rrArbFrrT)rbFrrTrl)rb)ErXrGtypingrrrrrrr r\torch.nnr-torch.nn.functional functionalrK timm.datar r timm.layersr rrrrr_builderr _featuresr _manipulater _registryrr__all__rnrraryrrrrmrrYr[rrrrZrrrrryr default_cfgsrrrrrrrrrrrrr;r:rsa* DDD AZZ*+#< (LryyL^@ @F> >B.")).b;RYY;|?7?7D ## # # #>E#s(O>%>>E#sTWY\J\D]>B;;|D "2222S 2  2  22222222]]2v "))))S )  )  )))))]])Xj299jZ#434*c#$sCx.(%! Z" Z0" Z" Z0" Z" Z0" Z" [-1" [" Z-1" [-1S-&- `DCDDCDDCDDCDDCDDCDDCDDCD D C D   D C D   D C D  r;