kh 4dZddlZddlZddlmZmZmZddlmZddl m Z m Z m Z m Z mZmZmZmZmZddlZddlmZddlmcmZddlmZddlmZmZddlmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%dd l&m'Z'dd l(m)Z)dd l*m+Z+m,Z,dd l-m.Z.m/Z/dd l0m1Z1m2Z2m3Z3ddl4m5Z5m6Z6ddgZ7ejpe9Z:eGddZ;de;de;fdZ< d@dejzdee>e>fde?deejzee>e>fffdZ@dejzfdZAe,GddejZCe+dddejfdejzd e>d!e?d"e e>d#ejde ejzf d$ZFe+ dAdejzde ejzd%eGd e>d&e?dejzf d'ZHGd(dejZIdBd)eGd*eJde fd+ZKd,e eGefd-eIde eGeffd.ZLdCd/eGde eGeffd0ZMe/eMd12eMd12eMd12eMeMeMd3ZNdDd4eGd5e?deIfd6ZO dDd4eGd5e?deIfd7ZPe.dDd5e?deIfd8ZQe.dDd5e?deIfd9ZRe.dDd5e?deIfd:ZSe.dDd5e?deIfd;ZTe.dDd5e?deIfd<ZUe.dDd5e?deIfd=ZVe.dDd5e?deIfd>ZWe.dDd5e?deIfd?ZXy)Ea NaFlex Vision Transformer An improved version of the Vision Transformer with: 1. Encapsulated embedding and position encoding in a single module 2. Support for linear patch embedding on pre-patchified inputs 3. Support for NaFlex variable aspect, variable resolution 4. Support for FlexiViT variable patch size 5. Support for NaViT fractional/factorized position embedding Based on ideas from: - Original Vision Transformer: https://arxiv.org/abs/2010.11929 - FlexiViT: https://arxiv.org/abs/2212.08013 - NaViT: https://arxiv.org/abs/2307.06304 - NaFlex (SigLip-2): https://arxiv.org/abs/2502.14786 Hacked together by / Copyright 2025, Ross Wightman, Hugging Face N) dataclassfieldsreplace)partial) CallableDictListOptionalSetTupleTypeUnionAny) pad_sequenceIMAGENET_INCEPTION_MEANIMAGENET_INCEPTION_STD)AttentionPoolLatentMlp to_2tuple get_act_layerget_norm_layer LayerNorm_assert)build_model_with_cfg)feature_take_indices)register_notrace_functionregister_notrace_module)register_modelgenerate_default_cfgs) checkpointcheckpoint_seq named_apply)Blockglobal_pool_nlc NaFlexVitCfg NaFlexVitczeZdZUdZdZeeeeeffed<dZ eed<dZ eed<dZ eed<d Z e ed <d Zeed <d Zeed<d Zeed<dZe ed<dZee ed<dZe ed<dZe ed<dZe ed<dZe ed<dZe ed<d Zeed<dZeed<dZeed<dZeeeefed<d Zeed!<d Z eed"<d Z!eed#<d Z"eed$<d Z#eed%<d Z$eed&<dZ%eeed'<d(Z&eed)<d Z'eed*<d+Z(eed,<d Z)eed-<d.Z*eed/<dZ+eeed0<dZ,eeed1<dZ-eeed2<dZ.eeed3<dZ/eeed4<dZ0eeed5<d Z1eed6<y)7r'zConfiguration for FlexVit model. This dataclass contains the bulk of model configuration parameters, with core parameters (img_size, in_chans, num_classes, etc.) remaining as direct constructor arguments for API compatibility.  patch_size embed_dim depth num_headsg@ mlp_ratioTqkv_biasFqk_norm proj_biasattn_drop_rateN init_values drop_rate pos_drop_ratepatch_drop_rateproj_drop_ratedrop_path_rate class_tokenr reg_tokenslearned pos_embed)r*r*pos_embed_grid_sizebicubicpos_embed_interp_modepos_embed_ar_preservingpos_embed_use_grid_sampledynamic_img_padpre_norm final_normfc_normmap global_poolpool_include_prefix weight_initfix_initlinearembed_proj_typeinput_norm_layerembed_norm_layer norm_layer act_layerblock_fn mlp_layerenable_patch_interpolator)2__name__ __module__ __qualname____doc__r+rintr __annotations__r-r/r0r1floatr2boolr3r4r6r7r r8r9r:r;r<r=r>r@strrArCrDrErFrGrHrIrKrLrNrOrQrRrSrTrUrVrWrXQ/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/models/naflexvit.pyr'r'6s/1Jc5c?*+0IsE3OIsIuHdGTItNE$(K%'IuM5 OU NENEKJIs5=%S/2=!*3*$)T)&+t+"OT!HdJ"GXd^"K %%KHd$OS#&*hsm*&*hsm*!%J $#Ix}#"Hhsm"#Ix}#',t+rccfgreturnc t|jj}|jDcic] \}}||vs ||}}}|r t |fi|}|Scc}}w)zIOverlay kwargs onto config, replacing config values with provided kwargs.)set__dataclass_fields__keysitemsr)rekwargs config_fieldskv config_kwargss rd_overlay_kwargsrq{sb005578M&,llnKda]8JQTKMKc+]+ J Ls AATxr+padcZ|j\}}}}|\}}|r@||zdk7s||zdk7r0|||zz |z} |||zz |z} tj|d| d| f}||z||z} } |j||| || |j ddddddj || | z||z|z} | | | ffS)ahPatchify a batch of images. Args: x: Input tensor of shape [B, C, H, W]. patch_size: Patch dimensions (patch_h, patch_w). pad: Whether to pad images to be divisible by patch size. Returns: Tuple of (patches, grid_size) where patches has shape [B, N, P*P*C] and grid_size is (num_patches_h, num_patches_w). rr$)shapeFrsviewpermutereshape)rrr+rsBCHWphpwpad_hpad_wnhnwpatchess rdbatch_patchifyrs JAq!Q FB B! q2v{a"f"a"f" EE!a5) * "Wa2gBffQ2r2r*221aAq!DLLQPRUWPWY[^`Y`cdYdeG RH rc_coordc (|dddddfjddz}|dddddfjddz}t||Dcgc]7\}}t|jt|jf9c}}Scc}}w)Nrr$dim)amaxzipr]item)rmax_ymax_xhws rdcalculate_naflex_grid_sizesrs~ 1a7O Q '! +E 1a7O Q '! +E7:5%7H Itq!S]CM * II IsrFdefault_img_sizer@rArCrDrErRproj_norm_layerrTr9r:rXrfct|||_||_| |_| |_||_t||_||_ ||_ ||_ ||_ |rdnd|_ |xj|z c_ |r*tjt!j"dd|nd|_|r*tjt!j"d||nd|_d|_d|_| | |_nV| Tt| |_t-t/|j(|jDcgc] \}}||z c}}|_|dk(rx|jd|jdz|z}|dur |Jd|dur|n|xsd}|r||nd|_tj2||||_d|_d|_n8|rJd|_tj:||||| |_d|_d|_|jr#dd lm}||j||| d |_ nd|_ |dur |Jd |dur|n|xsd}|r||ntjB|_"| d vr|j* tGdd|_$d|_%d|_&| r| dk(rd|_'n| dk(rd|_'n| dk(r|j*J|j*\}}d|_'tjt!jPd||dz|_%tjt!jPd||dz|_&nW|j*J|j*\}}tjt!jPd|||dz|_$d|_'tjR||_*|dkDr ddl+m,}|||j|_-ytjB|_-ycc}}w)aInitialize NaFlexEmbeds module. Args: patch_size: Size of patches for patch embedding. in_chans: Number of input image channels. embed_dim: Dimensionality of patch embedding. proj_type: Type of embedding projection layer ('conv' or 'linear'). proj_bias: Whether to use bias in projection layers. class_token: Whether to include a class token. reg_tokens: Number of register tokens to include. dynamic_img_pad: Whether to enable dynamic padding for variable resolution. default_img_size: Default image size for position embedding grid calculation. pos_embed: Type of position embedding ('learned', 'factorized', 'rope', 'none'). pos_embed_grid_size: Grid size for position embedding initialization. pos_embed_interp_mode: Interpolation mode for position embedding resizing. pos_embed_ar_preserving: Whether to preserve aspect ratio during interpolation. input_norm_layer: Normalization layer applied to input (linear mode only). proj_norm_layer: Normalization layer applied after projection. norm_layer: Default normalization layer. pos_drop_rate: Dropout rate for position embeddings. patch_drop_rate: Dropout rate for patch tokens. enable_patch_interpolator: Enable dynamic patch size support. r$rNrPTz5`norm_layer` must be given when input_norm_layer=True)biasF) kernel_sizestrider)PatchEmbedInterpolator)base_patch_sizerr- interpolation antialiasz4`norm_layer` must be given when proj_norm_layer=True) factorizedr?zgCannot initialize position embeddings without grid_size.Please provide img_size or pos_embed_grid_size.noneroperg{Gz?r?)p) PatchDropout)num_prefix_tokens).super__init__has_class_tokennum_reg_tokensrCrDrErr+rr-rFrXrnn Parametertorchzeros cls_token reg_tokenrrAtupler norm_inputLinearprojflatten is_linearConv2d timm.layersrpatch_interpolatorIdentitynorm ValueErrorr@ pos_embed_y pos_embed_xpos_embed_typerandnDropoutpos_droptimm.layers.patch_dropoutr patch_drop)selfr+rr-rr4r=r>rFrr@rArCrDrErRrrTr9r:rXsr patch_dimrrrr __class__s rdrzNaFlexEmbeds.__init__s\ *(%:"'>$)B&#J/  ".)B&'2q *,HSekk!Q &BCX\PZekk!Z&KL`d<@>B  *':D $  )$-.>$?D !',TEZEZ\`\k\kAl-mAa1f-m'nD $  *T__Q-??(JI(D0Z5G HG HH-=-EzL\Ld`d =M.y9SWDO )YYGDI DL!DN( ''"DO )JU^DI DL"DN  ) ) :&< $!#3 'D #'+D #$t+ 0B C B CC(74(?*oF]Y]2AOI.r{{}  1 1d6N6N6VBC C263737I/"(D  & "(D  , &++7 77++DAq".D !||EKK1i,H3,NOD !||EKK1i,H3,NOD ++7 77++DAq\\%++aAy*IC*OPDN"+D  ]3 Q  >*"&"8"8DO !kkmDO[.ns7P cDt|j|jS)zGet feature information for feature extraction. Args: location: Feature extraction location identifier Returns: Dictionary containing feature channel count and reduction factor )num_chs reduction)dictr-r+)rlocations rd feature_infozNaFlexEmbeds.feature_infossDNNdooFFrc as_scalarcH|rt|jS|jS)zGet the feature reduction ratio (stride) of the patch embedding. Args: as_scalar: Whether to return the maximum dimension as a scalar Returns: Feature reduction ratio as scalar or tuple )maxr+)rrs rd feat_ratiozNaFlexEmbeds.feat_ratio~s  t' '?? "rcimg_sizec|jrPtj|d|jdz tj|d|jdz fS|d|jdz|d|jdzfS)aCalculate grid (feature) size for given image size. Takes into account dynamic padding when enabled. Args: img_size: Input image size as (height, width) Returns: Grid size as (grid_height, grid_width) rr$)rFmathceilr+)rrs rddynamic_feat_sizezNaFlexEmbeds.dynamic_feat_sizes   99Xa[4??1+==> (ST+X\XgXghiXjJj@kk kA;$//!"44hqkT__UVEW6WW Wrcrr patch_coordc z t|}jjdd\ jjddddj fd}i}t |D]&\}}|j |gj|(|jD]\}}||} tjd| jd} ddd| fjdtj|j| ddd| fjt|ddy) a_Apply learned position embeddings to NaFlex batch in-place. Interpolates learned 2D position embeddings for each sample in the batch based on their individual grid sizes. Args: x: Input tensor to add position embeddings to [B, N, C] patch_coord: Patch coordinates [B, N, 2] with (y, x) values r$rwrruc|dk(r)|dk(r!jjdzd}nzjrtt |n|}t j |jddddddd|dd|dfjdjdd}|jj S) a Return a flattened positional-embedding grid at an arbitrary spatial resolution. Converts the learned 2-D table stored in NCHW format (pos_embed_nchw) into a (1, H*W, C) sequence that matches the requested size. rr$FTsizemode align_cornersrNrudtype) r@r}rDrrrz interpolaterCr transposetor)rpos_embed_flat _interp_sizeorig_horig_wpos_embed_nchwrrrs rd _interp2dz?NaFlexEmbeds._apply_learned_naflex_pos_embed.._interp2dsQ6!Q6(9!%!7!76F?B!O7;7S7SyT3Y] !""%33"'" " Qa(47(* ", -4GAJyyA "$$177$3 3rcNdevicer)rr@ryr|r_ enumerate setdefaultappendrkmin index_add_r as_tensorrexpandlen)rrrrnaflex_grid_sizesrsize_to_indicesbirn batch_indicesrseq_lenrrrs`` @@@rd_apply_learned_naflex_pos_embedz,NaFlexEmbeds._apply_learned_naflex_pos_embeds.8 D--a2//1a;AAC 4 4*=?01 9EB  & &q" - 4 4R 8 9!0 5 5 7  A}'q\N!''!*n&:&:1&=>G a'kN % % ahh?q(7({+223}3Er2N  rcc|j}|j\}}}|jdjdz}|jr.|j d}|j } | x} } | |z x} } n-|j d\} } | |dddfz } | |dddfz } t j|dd|t j}| |ddddf<| |ddddf<| dz |ddddf<| dz |ddddf<tj|||| | fd }tj|jjddddj|d d d j||j dd j#|j$ }t j&||j)d}|||dd|d|dfz }y)aApply learned position embeddings to NaFlex batch using grid_sample. Uses F.grid_sample for efficient interpolation of learned 2D position embeddings based on patch coordinates. Based on proposal by https://github.com/stas-sl Args: x: Input tensor to add position embeddings to [B, N, C] patch_coord: Patch coordinates [B, N, 2] with (y, x) values r$rrNrurw)rrFrrborderrr padding_moderr.r.r$)rryrvaluesrDrrrfloat32rz affine_grid grid_sampler@r|rr_rCrrarange unsqueeze)rrrrrr~NrshapesL_iL_global grid_size_y grid_size_xscale_xscale_ythetagridr@rs rd+_apply_learned_naflex_pos_embed_grid_samplez8NaFlexEmbeds._apply_learned_naflex_pos_embed_grid_samples''1aQ'..2  ' '++!+$CxxzH(0 0K+ (3 .Gg'-{{q{'9 $K!F1a4L0G!F1a4L0G Aq!F%--H aAg aAg 1aAg 1aAg}}UQ; $DTYZMM NN " "1aA . 5 5aR D J J L ++!  "177"  \\!F + 5 5a 8 Yr1k&1;v3FF GGrc grid_sizec8|jjdd\}}|d|k(r)|d|k(r!|jjd||zd}n|jrt |}||f}n|}t j |jjddddj||jddd d d d d |dd |dfjdjdd}|j|j }|j|y ) a0Apply learned position embeddings to standard 2D batch in-place. Interpolates learned 2D position embeddings to match the specified grid size. Args: x: Input tensor to add position embeddings to [B, H*W, C] grid_size: Target grid size as [height, width] r$rwrrruFTrNr)r@ryr}rDrrzrr|r_rCrrrradd_)rrrrrrrLrs rd_apply_learned_pos_embedz%NaFlexEmbeds._apply_learned_pos_embeds--a2 Q<6 !ilf&<!^^33AvKN++ N !t ( ]]&&q!Q288:!//#  MYq\M=IaL=0 2 3:'!*YYq!_ (***9 ~rcc t|}t|jdk(sJ|jjd|j jd}}i}t |D]&\}}|j|gj|(dtjdtdtdtjffd } |jD]\}} |\} } |jrt| | x} }n| | }} | |j| |ddd| f}| |j ||ddd| f}|jd |jdz}|j!dd }t#jd|jd}ddd|fj%dtj&| j( |ddd|fj+t| d d  y) acApply factorized position embeddings to NaFlex batch in-place. Uses separate Y and X position embedding tables that are interpolated and combined for each sample's grid size. Args: x: Input tensor to add position embeddings to [B, N, C] patch_coord: Patch coordinates [B, N, 2] with (y, x) values rr$table new_length orig_lengthrfc ||k(r|jjStj|j dddj |ddj dddjjS)z Resample a 1-D positional-embedding table to specified length and return it in (1, L, C) layout, dtype matching x. rrrur$rPFrrrrrrzrr|r_rrrrrs rd _interp1dzBNaFlexEmbeds._apply_factorized_naflex_pos_embed.._interp1d@st [(xxaggx..== aA&,,.#  gaArrr0  1rcNrurr)rrrrryrrrrrTensorr]rkrDrr rrrrrr)rrrrrrrrrrnr#rtarget_htarget_wlen_ylen_xpe_ype_xposrs ` rd"_apply_factorized_naflex_pos_embedz/NaFlexEmbeds._apply_factorized_naflex_pos_embed&s8 D$%222))//2D4D4D4J4J14M=?01 9EB  & &q" - 4 4R 8 9 1U\\ 1s 1 1QVQ]Q] 1!0 5 5 7  A}!" Hh++ #Hh 77'uT--uf=a(lKDT--uf=a(lKD..#dnnQ&77C++a#C!''!*ciil3G a'kN % % ahh?AxxK ''M(:BC  rccj}j\}|jddz}|jr.|jd}|j}|x}} ||z x} } n,|jd\}} | |dddfz } ||dddfz } dtj dtj dtj dtj ffd } | |j | | } | |j| | }t j| jd}| |ddd|d f||ddd|d fzz y)aApply factorized position embeddings to NaFlex batch using grid_sample. Uses F.grid_sample for efficient interpolation of separate Y and X position embedding tables based on patch coordinates. Based on proposal by https://github.com/stas-sl Args: x: Input tensor to add position embeddings to [B, N, C] patch_coord: Patch coordinates [B, N, 2] with (y, x) values r$rrNrscale out_lengthrfc|jdddjdjdddj}t j ddj }||ddddf<|dz |ddddf<d|ddddf<tj|d|fd}tj||d dd }|jjS) Nrrur$rrwrFrbilinearrr) r|r rr_rrrrzrrrr) rr.r/perrr~rrrs rdr#zNNaFlexEmbeds._apply_factorized_naflex_pos_embed_grid_sample.._interp1dsq!Q'11!4;;Ar2rJPPRBKK1a9E"E!Q'N"QYE!Q'NE!Q'N==Aq*(=USDr4j\deB55> !rc)r.r/rrr) rryrrDrr$rrr r )rrrrr_r r rrrrrr#r*r)rr~rs ` @@rd._apply_factorized_naflex_pos_embed_grid_samplez;NaFlexEmbeds._apply_factorized_naflex_pos_embed_grid_samplecsW''1a!!a!(1,  ' '++!+$CxxzH(0 0K+ (3 .Gg(.{{1~ $K!F1a4L0G!F1a4L0G "U\\ "%,, "ELL "]b]i]i "))[Q))[Q \\!F + 5 5a 8 T"aK// 04Aq+fBU8U3V VVrcc|jjd|jjd}}|\}}|jrt ||x}}n||}}dt j dtdtdt j ffd } | |j||ddd|f} | |j||ddd|f} | jd| jdz} | jdd} j| y) a]Apply factorized position embeddings to standard 2D batch in-place. Uses separate Y and X position embedding tables that are interpolated and combined for the specified grid size. Args: x: Input tensor to add position embeddings to [B, H*W, C] grid_size: Target grid size as [height, width] r$rrrrfc ||k(r|jjStj|j dddj |ddj dddjjS)Nrrrur$rPFr r!r"s rdr#z;NaFlexEmbeds._apply_factorized_pos_embed.._interp1dsr[(xxaggx..== aA&,,.#  gaArrr0  1rcNru) rryrrDrrr$r]r rr)rrrrrrr%r&r'r(r#r)r*r@rs ` rd_apply_factorized_pos_embedz(NaFlexEmbeds._apply_factorized_pos_embeds ))//2D4D4D4J4J14M&(  ' '(3 3EE#X5E 1U\\ 1s 1 1QVQ]Q] 1))5&9!YhY,G))5&9!YhY,GNN1%q(99 "**1a0 ~rcc d}|jd}|jr5|?t|jdk(dt ||j |j \}}n*t|jdk(xs|jdk(d|jr}|jdk(rnt|jdud |j||jj|jjt|jd dd }nC|jd }|j|j|}|j|}nt|jdk(d |j r|jdd\}}|j d||j dzz |j dz}|j d||j dzz |j dz}tj |d|d|f}|j|}|jdd}|jr!|jd j#dd }|j%|}|j&dk(rJ||j)||n|j*r|j-||n|j/||no|j&dk(rJ||j1||nJ|j*r|j3||n*|j5||n|j&dk(rJdg} |j6,| j9|j6j;|dd|j<,| j9|j<j;|dd| rt?j@| |gzd}|jC|}|jE|}|S)a6Forward pass for patch embedding with position encoding. Args: x: Input tensor. Supported formats: - [B, C, H, W] for conv mode - [B, N, P*P*C] for pre-patchified linear mode (normal) - [B, N, Ph, Pw, C] for pre-patchified linear mode (variable patch size) patch_coord: Optional patch coordinates [B, N, 2] for NaFlex mode. Returns: Embedded tensor with position encoding and class/register tokens. Shape: [B, num_prefix_tokens + N, embed_dim] Nrrvz-Expecting 2D image input with input ndim == 4)rsrxrwz/Expecting patchified input with ndim == 3 or 5.z,input norm not supported with patch resizingruT)r+rzConvolutional input must be 4Dr$r?)rrrrzROPE not yet implementedrr)#ryrrndimrr+rFrXrrrweightrrrrzrsrrrrrErrr7r4r,rrrrrcatrr) rrrrrr~rrrrto_cats rdforwardzNaFlexEmbeds.forwardsl$*. GGAJ >>"! %TU-adFZFZ[ 9! 2qvv{4ef--!&&A+4/1_`++II$$IINN$QWWQq\2" ,IIaL??.*AIIaL AFFaK!A B##wwrs|1+a$//!2D.DDXYHZZ+a$//!2D.DDXYHZZEE!a512 ! A I||IIaL**1a0 IIaL   ) +$--a9-E11DDQT_D`88 8T  L 0$00i0H11GGWbGc;;A;;W  F * 44 45 >> % MM$..//2r: ; >> % MM$..//2r: ;  &A3,A.A MM!  OOA rc)r*rwr,NTTrFNr?)r@rBFFNNNr5r5FTN)rYrZr[r\rr]r r rar`r rModuler_rrrrrrrr$rrr rr,r4r7r? __classcell__rs@rdrrs>'V79 '+" $$)FJ&=E)2,1.3:>FJ48#%%'.3+X,c5c?23X,X, X,  } X,  X,X,X,"X,'uS%S/-A'BCX,X,"*%S/!:X,$'X,&*X,(,X, 'tBII7!X,"#4$ryy/)B#BC#X,$!bii1%X,&!'X,(#)X,*(,+X,, -X,t GS#X G #D #E#uS#X:N4O #X%S/XeCHoX"6||66  6p,H||,H,H  ,H\!||!Cy!  !F;||;;  ;z-W||-W-W  -W^(||(Cy(  (Z37f||f"%,,/f  frcr patch_validr symmetricq_lenrc |y|j}|j\}}|}|dkDrA|j||ftj}tj||gd}||z }|r5|j d|j dz} | j d} n%|xs|}|ddddddfj |d||} tj| |} | j| tj|j| S)a3Creates an attention mask from patch validity information. Supports two modes controlled by `symmetric`: 1. `symmetric=True` (default): Creates a symmetric mask of shape [B, 1, seq_len, seq_len]. An attention pair (i, j) is allowed only if both token i and token j are valid. Suitable for standard self-attention. 2. `symmetric=False`: Creates a potentially non-square mask of shape [B, 1, q_len, kv_len]. An attention pair (q, k) is allowed only if the key/value token k is valid. Query token validity is not checked in the mask itself. Useful for cross-attention or specific self-attention implementations `q_len` can be specified. Used for NaFlex mode to handle variable token counts and padding tokens. Args: patch_valid: Tensor of shape [B, N] with True for valid patches, False for padding. num_prefix_tokens: Number of prefix tokens (class token, register tokens) to prepend, which are always considered valid. symmetric: If True, create a symmetric mask. If False, create an expanded mask based only on key/value validity. q_len: Query sequence length override. Only used when `symmetric` is False. Defaults to the key/value sequence length (`kv_len`) if None. dtype: Dtype of the output attention mask (e.g., torch.float32). Returns: Attention mask tensor. Additive mask (-inf for masked, 0 for unmasked). Shape is [B, 1, seq_len, seq_len] if symmetric=True, or [B, 1, q_len, kv_len] if symmetric=False. Nrrr$rr) r`rynew_onesrr=r r zeros_like masked_fill_finfor) rFrrGrHrr~r kv_len prefix_valid mask_bool mask_floats rdcreate_attention_maskrR%sJ""$K   DAq F1"++Q0A,B%**+U ii{ ;C ##))"- 0E0Ea0HH ''* 4q 0188AufM !!)59JYJ E(:(>(>? rc pool_typereduce_include_prefixc||dvrt||||}|S|dkDrF|r9|j|jd|}tj||gd}n |dd|df}|j |j }|dk(rN||jdzjd}|jdd jd }||z } | S|d k(r||jdzjd}|jdd jd }||z } |j} tj| j j| |<| jd} d | | zzS|dk(rO|j} tj| j j| |<| jdSJ)aPGlobal pooling with NaFlex support for masked tokens. Applies global pooling while respecting patch validity masks to exclude padding tokens from pooling operations. Args: x: Input tensor with shape [B, N, C] patch_valid: Optional validity mask for patches [B, N-num_prefix_tokens] pool_type: Type of pooling ('token', 'avg', 'avgmax', 'max') num_prefix_tokens: Number of prefix tokens (class/register) reduce_include_prefix: Whether to include prefix tokens in pooling reduction Returns: Pooled tensor with shape [B, C] N)avgavgmaxrrSrrTrr$rrVrT)rkeepdim)rrWg?r)r&rJryrr=rrr sumclampclonerMrr) rrrFrSrrTrOpatch_valid_float masked_sums valid_countspooled masked_avgmasked_x masked_maxs rdglobal_pool_naflexrdjs.i/GG  /"7   1 '// d |j@ d |j d |j0d|d|jBd|j d|jDd|jFd|jHd|jJd|d|jLd|jNdtQ|dd|_)|j@r||j(ntjT|_+tYjZd|j\|j^D cgc]} | ja} } tjbte|j^D cgc]t} | |j(|jf|jh|jj|jl|jn|jp|jr|jt| | |||  vc} |_;|jRjyd}te|j^D cgc]} t{d| |j(|c} |_>|j~r|js||j(ntjT|_A|jdk(r4t|j(|jf|jh|||_Cnd |_C|j}||jd!k(}|j~r|r||j(ntjT|_@tj|j|_F|dkDr tj|j(|ntjT|_H|jd"k7r|j|j|jr|jy y cc} wcc} wcc} w)$aInitialize NaFlexVit model. Args: cfg: Model configuration. If None, uses default NaFlexVitCfg. in_chans: Number of input image channels. num_classes: Number of classification classes. img_size: Input image size (for backwards compatibility with classic vit). **kwargs: Additional config parameters to override cfg values. rMrVrWrtokenrJri)rMrr?rr$rFr+rr-rr4r=r>rrFr@rArCrDrErr9r:rX) rr0r1r2r3r4r7 proj_drop attn_drop drop_pathrTrUrWT)rzblocks.)modulerrrJ)r0r1rTrUNrVskiprb)Mrrr'rqrKr=r@rrTrrSrrUrGELUrVr%rWrrfr- num_featureshead_hidden_sizerr>rrrLgrad_checkpointingrr+rQrGrFrArCrDrEr9r:getattrembedsrnorm_prerlinspacer<r/r Sequentialranger0r1r2r3r4r7r;r6blocksrrrrHrIrr attn_poolrr8 head_droprheadrN init_weightsrOfix_init_weight)rrerrfrrlrTrSrUrVrWrrdpripatch_reductionrIrs rdrzNaFlexVit.__init__s" #\^ !#00C"NNNN#//W"<<<}} EEEE$CNN3@y )#*>*>?!#--0;BGG <<(5MM(S '??EH]]RRD1DN&)oo1 #..0!nn"#&#:#: "'# ~~  mm ))  ,,&    ~~ &  // mm !$ 7 7 #&";"; %($?$? '*&C&C - ++! " //# $'.c3NPU&V%  (69\\ 3==1r{{} "'33E3Esyy!QRAqvvxRRmm 399%!&'  MM---- --OO,,,,a&%## &'( &++0040@399%  '! s}} X  25 Js}}-Y[YdYdYf  ??e #0----%# DN"DN++ ?oo.G47NNwz#--0TVT_T_Ta CMM2>IAoBIIdnnk:SUS^S^S` ??f $   coo . <<  " eS&'( sU:A9U?"VcFdtjdtddfd}t|jD]m\}}||j j jj|dz||jjjj|dzoy)z8Apply initialization weight fix with layer-wise scaling.param _layer_idrfNcR|jtjd|zy)Ng@)div_rsqrt)rrs rdrescalez*NaFlexVit.fix_init_weight..rescale?s JJtyyy1 2rcr$) rr$r]rryattnrr<datamlpfc2)rrlayer_idlayers rdr~zNaFlexVit.fix_init_weight=s 35<< 3C 3D 3 )5 =OHe EJJOO**//A > EIIMM((--x!| < =rcrc|dvsJd|vr tj|j nd}tt |||y)zInitialize model weights according to specified scheme. Args: mode: Initialization mode ('jax', 'jax_nlhb', 'moco', or '') )jaxjax_nlhbmocorMnlhbr5N)rlogrfr#get_init_weights_vit)rr head_biass rdr}zNaFlexVit.init_weightsFsD 666639T>TXXd..//r (y94@rccheckpoint_pathprefixc2ddlmdfd }||||y)Nr$) _load_weightsctj|d}t|tr d|vr|d}t |j D]}|j dr|j||d|z<,|j dr|j||d|z<U|j dr|j||d|z<~|j ds|j||d|d d z<|||S) z8Adapter function to handle the different model structurecpu) map_location state_dictrembeds.rr@ patch_embedr.N)rload isinstancerlistrj startswithpop)modelrrrrn_orig_load_weightss rd_load_weights_adapterz8NaFlexVit.load_pretrained.._load_weights_adapterUsO%HJ*d+ 0J' 5 *//+, G<< ,0:q0AJy1}-\\+.0:q0AJy1}-\\+.0:q0AJy1}-\\-05?^^A5FJy1RS612 G&eZ@ @rcrM)vision_transformerr)rrrrrs @rdload_pretrainedzNaFlexVit.load_pretrainedPs L A& dOVembeds.cls_tokenembeds.pos_embedembeds.reg_tokenrb)r skip_lists rdno_weight_decayzNaFlexVit.no_weight_decayjsQ rccoarsec tdddgS)zGet parameter group matcher for optimizer parameter grouping. Args: coarse: Whether to use coarse-grained grouping Returns: Dictionary mapping group names to regex patterns z^embeds)z^blocks\.(\d+)N)z^norm)i)stemry)r)rrs rd group_matcherzNaFlexVit.group_matcherts-/CD  rcenablec||_t|jdrGt|jjdr&|jjj |yyy)zEnable or disable gradient checkpointing for memory efficiency. Args: enable: Whether to enable gradient checkpointing rset_grad_checkpointingN)rrhasattrrtrr)rrs rdrz NaFlexVit.set_grad_checkpointingsO#) 4;; .74;;;R;RTl3m KK # # : :6 B4n .rcc|jS)zeGet the classification head module. Returns: Classification head module )r|)rs rdget_classifierzNaFlexVit.get_classifiersyyrcrKc||_|=|dvsJ|dk(r|jJd|dk7r|jd|_||_|dkDr&tj|j ||_ytj |_y)zReset the classification head with new number of classes and pooling. Args: num_classes: Number of classes for new classification head global_pool: Optional new global pooling type NrhrJz=Cannot currently add attention pooling in reset_classifier().r)rfrzrKrrr-rr|)rrfrKs rdreset_classifierzNaFlexVit.reset_classifiers'  ""NN NNe#(>]]]u%$..*D!%*D >IAoBIIdnnk: SUS^S^S` rcrrindicesreturn_prefix_tokensr stop_early output_fmtintermediates_only output_dictrrF attn_maskc |dvsJd|dk(} g} tt|j|\}}t|tr|d} |d} |d}Jd|}|j dd \}}|j j||f\}}| #| !t| |j|j} |j || }|j|}tjjs|s |j}n|jd |d z}t|D]z\}}| ||| }n?|j r+tjjs t#||}n||}||vsW| j%|r|j'|n|||jrE| Dcgc]}|d d d |jf}}| Dcgc]}|d d |jd f} }nd }| rN| Dcgc]C}|j)|j d ||dj+d dd dj-E} }|r*i}| |d<||r||d<|s|j'|}||d<|Stjjs|r|t/t1| |} |r| S|j'|}|| fScc}wcc}wcc}w)an Forward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if int, all if None, select matching indices if sequence return_prefix_tokens: Return both prefix and spatial intermediate tokens norm: Apply norm layer to all intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features output_dict: Return outputs as a dictionary with 'image_features' and 'image_intermediates' keys patch_coord: Optional patch coordinates [B, N, 2] for NaFlex mode patch_valid: Optional patch type indicators (1=patch, 0=padding) for NaFlex attn_mask: Optional attention mask for masked attention Returns: A tuple with (final_features, intermediates), a list of intermediate features, or a dictionary containing 'image_features' and 'image_intermediates' (and optionally 'image_intermediates_prefix') )NCHWNLCz)Output format must be one of NCHW or NLC.rrrFrzWIP, patch mode needs more workr9Nr:r$rrrrwruimage_intermediatesimage_intermediates_prefiximage_features)rrryrrryrtrrRrrrurjit is_scriptingrrrr!rrr}r| contiguousrr)rrrrrrrrrrrrFrr} intermediates take_indices max_indexrheightwidthrrryrblky prefix_tokens result_dictx_finals rdforward_intermediateszNaFlexVit.forward_intermediatessF_,Y.YY,& "6s4;;7G"Q i a M*KM*K lG ;; ;5GGGBCLMFE;;00&%ADAq  !8-k4;Q;QSZS`S`aI KK[K 9 MM!  99 ! ! #:[[F[[)a-0F' BFAs$Y/((1G1G1IsA&FL $$TTYYq\qA B  ! !ERSQq!D$:$:"::;SMSDQRqQq$"8"8"99:RMR M ' !''!*aB/771aCNNPM K1>K- .(-AK8AKc|"t||j|j}|j||}|j |}||j D] }|||}nR|j r5tjjst|j |}n|j |}|j|}|S)Nrrr:r) rRrrrtruryrrrrrr"r)rrrrrFrrs rdforward_featureszNaFlexVit.forward_featuress  -"&"8"8ggI KK{K 3 MM!   {{ 0Y/ 0  $ $UYY-C-C-Et{{A.A AA IIaLrcrScV|jht||jr |jnddd|j}|js|dd|jdf}|j||}|S| |j n|}t ||||j|j}|S)NrFr$)rrGrHrrrX)rzrRrLrrrKrd)rrrrSrFrs rd_poolzNaFlexVit._pool<s >> %-<@B {#,'{#{# {# uS%S/%9:; {# {#z=AAdA YY=s=C===2 YY YY  D  T     YYCTCTCC YY aCahsmaW[a(8<).$$', %262604s U\\4U\\(9#::;s eCcN34s #' s  s  s s !%s s "%,,/s "%,,/s   -s  tELL!5tELL7I)I#JDQTVYQYNZ [s p372604 ||"%,,/"%,,/    -   D(,26 || }"%,,/   B %26 1|| 1 1"%,,/ 1  13726 :U\\4U\\(9#::;:"%,,/:"%,,/ :  :rcrrcHddlm}m}m}d|vr t ||Sd|vr|S|S)zFFunction imported from vision_transformer.py to maintain compatibilityr$)init_weights_vit_jaxinit_weights_vit_mocoinit_weights_vit_timmr)rr)rrrrr)rrrrrs rdrrs0ff }+yAA 4$$$$rcrrcddlm}i}|jD]Y\}}|dk(rt|jdr|j dk(rd}d}d|vr|dj d}d|vr|dj d}||z}|j d} | |z } tj| } tj| } | | k7r| jro| js_| } |ddd|f} | jr |dxx| z cc<|dd||f}|jr |dxx|z cc<|dd|df}| } t| } | | z| k(r"|jd| | |j d }nt|jjd rl|jjj\}}||z| k(r"|jd|||j d }ntj!d | d ||zd ||d<|dk(r||d<|dk(r||d< |j#dr9|dd}|dk(r#|j%dd ddj'd}d|z}|||<U|||<\|S)zZHandle state dict conversion from original ViT to the new version with combined embedding.r$)checkpoint_filter_fnr@rwrrrNrurz-Position embedding size mismatch: checkpoint=z, model=z?. Using default initialization and will resize in forward pass.rrrz patch_embed.r.z proj.weightr)rrrkrrtr;ryrr is_integernumelr]r}rr_loggerwarningrr|r)rrorig_filter_fnout_dictrnro num_cls_token num_reg_tokenr num_patchesnum_patches_no_prefixgrid_size_no_prefixr cls_token_emb reg_token_embrrsuffixnew_keys rdrrsJH  "A1 u||[1affk ! ! *,$.{$;$A$A!$DM*,$.{$;$A$A!$DM$1M$A! ggaj (36G(G%&*ii0E&F# IIk2 '940;;=iFZFZF\#8K$%a=&8$9M$**,";/=@/$%a})D&D$EM$**,";/=@/!.//0A 3I N y(K7 !Y 1771:FAu||77E$||77AA1q5K/ ! !Q1771: >A$OO"OP[}\dfgjkfkdmn`!a ,-H' ( + +,H' ( + +,H' ( \\. )rsVF&IIaAq)11!4&(G !HW HQKCAF Orcurlc 2|dddddttdddd |S) Nr)rwrg?rBz embeds.projr|z apache-2.0) rrf input_size pool_sizecrop_pctrmeanstd first_conv classifierlicenser)rrls rd_cfgrs7#"'%#    rcztimm/) hf_hub_id)z)naflexvit_base_patch16_gap.e300_s576_in1kz-naflexvit_base_patch16_par_gap.e300_s576_in1kz0naflexvit_base_patch16_parfac_gap.e300_s576_in1kz$naflexvit_base_patch16_map.untrainedz'naflexvit_base_patch16_siglip.untrainedz)naflexvit_so400m_patch16_siglip.untrainedvariant pretrainedc ~|jdd}|jdt}ttDchc]}|j}}t |Dcic]}||vs||j|}}|r t |fi|}t t||ft|t|dd|} | Scc}wcc}w)N out_indicesrwregetter)r  feature_cls)pretrained_filter_fnre feature_cfg) rr'rnamerrqrr(rr) r r rlr refcfg_field_namesrn cfg_updatesrs rd_create_naflexvitrs**]A.K **ULN +C'-l';rVrvT)r+r-r/r0r7rKr>rIr re)naflexvit_base_patch16_gapr'rr rlrers rdr r XsC  C  ezWZ e^d eE Lrcc Jtddddddddd }td ||d |}|S) z]ViT-Base with NaFlex functionality, aspect preserving pos embed, global average pooling. r*r,r.rTrVrv) r+r-r/r0r7rDrKr>rIr)naflexvit_base_patch16_par_gapr!r"s rdr$r$jsF  $ C  i:[^ ibh iE Lrcc Ltdddddddddd }td ||d |}|S) zjViT-Base with NaFlex functionality, aspect preserving & factorized pos embed, global average pooling. r*r,r.rTrrVrv) r+r-r/r0r7rDr@rKr>rIr)!naflexvit_base_patch16_parfac_gapr!r"s rdr&r&}sI  $ C  lj^a lek lE Lrcc Ftddddddd}td ||d|}|S) zBViT-Base with NaFlex functionality and MAP attention pooling. r*r,r.rrJr$)r+r-r/r0r7rKr>r)naflexvit_base_patch16_mapr!r"s rdr(r(s@  C  ezWZ e^d eE Lrcc Ltddddddddd d }td ||d |}|S)/ViT-SO150M2 with NaFlex functionality for variable aspect ratios and resolutions. This model supports: 1. Variable aspect ratios and resolutions via patch coordinates 2. Position embedding interpolation for arbitrary grid sizes 3. Explicit patch coordinates and valid token masking r*@ NN@rFr$rVT) r+r-r/r0r1r7r2r>rKrIr)"naflexvit_so150m2_patch16_reg1_gapr!r"s rdr/r/sI  C  mz_b mfl mE Lrcc Jtddddddddd }td ||d |}|S) r*r*r+r,r-r.rFr$rJ) r+r-r/r0r1r7r2r>rKr)"naflexvit_so150m2_patch16_reg1_mapr!r"s rdr1r1sF  C  mz_b mfl mE Lrcc Dtdddddd}td||d|}|S) zGViT-Base with NaFlex functionality and SigLIP-style configuration. r*r,r. gelu_tanhrJ)r+r-r/r0rUrKr)naflexvit_base_patch16_siglipr!r"s rdr4r4s=   C  h*Z] hag hE Lrcc Ftddddddd}td ||d|}|S) zUViT-SO400M with NaFlex functionality for variable aspect ratios and resolutions. r*igZӼ @r3rJ)r+r-r/r0r1rUrKr)naflexvit_so400m_patch16_siglipr!r"s rdr7r7s@  C  jJ\_ jci jE LrcrA)Nrir$F)rr5rr)Yr\loggingr dataclassesrrr functoolsrtypingrrr r r r r rrrtorch.nnrtorch.nn.functional functionalrztorch.nn.utils.rnnr timm.datarrrrrrrrrrtimm.models._builderrtimm.models._featuresrtimm.models._features_fxrrtimm.models._registryrr timm.models._manipulater!r"r#rr%r&__all__ getLoggerrYrr'rqr$r]r`rrrCrrrrRrardr(r_rrr default_cfgsrrr r$r&r(r/r1r4r7rbrcrdrIs2$ 22OOO +E66WGKK6 ; ' '  H % A,A, A,H  L  <<#s(O 5<<sCx () @J Js 299s s l"##"]] A\\AAA} A {{ A ell AAH/3 !"&+ E <<Eell+EE E $ E  \\ EEPl l^ %s %u %x %IT#s(^III$sTWx.IXcT#s(^"%1526:69=9-1F/3v15& "s9*!&A&A&A &AR4i"t)$$Y& 4 i   4i04i. d     9  rc