kh~!ddlmZmZmZddlZddlmZddlmZddl m Z ddl m Z ddejd eejfd ZGd d ejZGd dejZy))FinalOptionalTypeN)nn) functional)use_fused_attn)apply_rot_embed_catscores attn_maskc||S||zSN)r r s Q/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/layers/attention.pymaybe_add_maskr s&6>FY,>>ceZdZUdZeeed< ddededededed ed ed ed e e e jd dffd Z ddejde ejd ejfdZxZS) AttentionaStandard Multi-head Self Attention module with QKV projection. This module implements the standard multi-head attention mechanism used in transformers. It supports both the fused attention implementation (scaled_dot_product_attention) for efficiency when available, and a manual implementation otherwise. The module includes options for QK normalization, attention dropout, and projection dropout. fused_attnNdim num_headsqkv_biasqk_norm scale_norm proj_bias attn_drop proj_drop norm_layerreturnc t |||zdk(sJd|s|r | Jd||_||z|_|jdz|_t |_tj||dz||_ |r| |jntj|_ |r| |jntj|_ tj||_|r| |ntj|_tj||||_tj||_y)agInitialize the Attention module. Args: dim: Input dimension of the token embeddings num_heads: Number of attention heads qkv_bias: Whether to use bias in the query, key, value projections qk_norm: Whether to apply normalization to query and key vectors proj_bias: Whether to use bias in the output projection attn_drop: Dropout rate applied to the attention weights proj_drop: Dropout rate applied after the output projection norm_layer: Normalization layer constructor for QK normalization if enabled rz$dim should be divisible by num_headsN>4==IQQRSUVXY[\^_`**Q-1a{{1~t{{1~1 ??..1a#.2mm$..**A DJJAq{{2r**D!$ 2D<>$'DqA KK1  % %aA . IIaL IIaL NN1 r)FFFTr8r8Nr)__name__ __module__ __qualname____doc__rbool__annotations__intfloatrrrModuler'torchTensorrO __classcell__r3s@rrrsd  "!$"!!48'/'/'/ '/  '/  '/'/'/'/!bii1'/ '/X15||  -  rrc eZdZUdZej j eed< dde de dedede de d e d e e d e e jd ed effd Z dde ej de ej fdZxZS) AttentionRopez A Self Attention module with ROPE support. Includes options for: * QK normalization option * Attention output (scale) normalization * Fused or unfused QKV projection support rrrr qkv_fusednum_prefix_tokensrr attn_head_dimrrrc Vt|| s| r | Jd||_||z} ||} | |jz} | dz|_||_t |_|r6tj|| dz||_ dx|_ x|_ |_ n^d|_ tj|| ||_ tj|| ||_ tj|| ||_ | r| | ntj|_| r| | ntj|_tj ||_| r| | ntj|_tj| ||_tj ||_y)aInitialize the Attention module. Args: dim: Input dimension of the token embeddings num_heads: Number of attention heads qkv_bias: Whether to add a bias term to the query, key, and value projections num_prefix_tokens: Number of reg/cls tokens at the beginning of the sequence that should not have position embeddings applied attn_drop: Dropout rate for attention weights proj_drop: Dropout rate for the output projection attn_head_dim: Dimension of each attention head (if None, computed as dim // num_heads) norm_layer: Normalization layer constructor to use for QK and scale normalization qk_norm: Enable normalization of query (Q) and key (K) vectors with norm_layer scale_norm: Enable normalization (scaling) of attention output with norm_layer Nr!r"r#r$)r&r'rr)rar rrr*r+q_projk_projv_projr,r-r.r/rr0r1r)r2rrrr`rarrrbrrrr(attn_dimr3s rr'zAttentionRope.__init__ksV:  ) i+i i)")#  $$Hdnn,% !2(* yyhlBDH6: :DK :$+ DH))CADK))CADK))CADK.5j*2;;= .5j*2;;= I.,6Jx(BKKM IIh, I.rroper c |j\}}}|jY|j|}|j||d|jdj ddddd}|j d\}} } n|j |j|||jdjdd}|j|j|||jdjdd} |j|j|||jdjdd} |j||j| } }||j} tj|ddddd| ddft|dddd| dddf|gdj!| }tj| ddddd| ddft| dddd| dddf|gdj!| } |j"r=t%j&|| | ||j(r|j*j,nd }nX||j.z}|| jd dz} t1| |} | j3d} |j+| } | | z}|jddj|||}|j5|}|j7|}|j9|}|S) aForward pass for the attention module. Args: x: Input tensor of shape (batch_size, sequence_length, embedding_dim) rope: Rotary position embeddings tensor for position-aware attention attn_mask: Optional attention mask to apply during attention computation Returns: Tensor of shape (batch_size, sequence_length, embedding_dim) Nr#r<r6rrr7r=r8r9r;)r>r+r?rr@rArdrFrerfr-r.rarZcatr type_asrrBrCrDrrEr)rrGr0r1r) r2r4rhr rHrIrJr+rKrLrMnptrNs rrOzAttentionRope.forwards ''1a 88 ((1+C++aAt~~r:BB1aAqQCjjmGAq! A&&q!T^^R@JJ1aPA A&&q!T^^R@JJ1aPA A&&q!T^^R@JJ1aPA{{1~t{{1~1  ((C 1Q4C4]+-@1aq=AQSW-XY_`aiijklA 1Q4C4]+-@1aq=AQSW-XY_`aiijklA ??..1a#.2mm$..**A DJJA B++D!$ 2D<>$'DqA KK1  % %aA . IIaL IIaL NN1 r) rPTTrr8r8NNFF)NN)rQrRrSrTrZjitrrUrVrWrXrrrrYr'r[rOr\r]s@rr_r_as %% !"%&!!+/*.!$7/7/7/ 7/  7/ # 7/7/7/$C=7/RYY7/7/7/x,004 55<<(5  - 5rr_r)typingrrrrZrtorch.nnrrBconfigr pos_embed_sincosr r[rrYrr_rrrrrs_(( $"1?5<<?HU\\4J?O OdxBIIxr