khdZddlmZddlZddlmZddlmcmZddl m Z m Z ddl m Z ddlmZdeefd ZGd d ej$ZGd d ej$Zy)aP Bottleneck Self Attention (Bottleneck Transformers) Paper: `Bottleneck Transformers for Visual Recognition` - https://arxiv.org/abs/2101.11605 @misc{2101.11605, Author = {Aravind Srinivas and Tsung-Yi Lin and Niki Parmar and Jonathon Shlens and Pieter Abbeel and Ashish Vaswani}, Title = {Bottleneck Transformers for Visual Recognition}, Year = {2021}, } Based on ref gist at: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 This impl is a WIP but given that it is based on the ref gist likely not too far off. Hacked together by / Copyright 2021 Ross Wightman )ListN) to_2tuplemake_divisible) trunc_normal_)_assert permute_maskc|j\}}}}||jddz}|jd|d|zdz }tj|ddgj d}tj|d|dz g}|jd|dzd|zdz }|ddd||dz df}|j||d||j dd|dd}|j|S)a Compute relative logits along one dimension As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 Args: q: (batch, heads, height, width, dim) rel_k: (2 * width - 1, dim) permute_mask: permute output dim according to this rrN)shape transposereshapeFpadflattenexpandpermute) qrel_kr BHWdimxx_pads W/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/layers/bottleneck_attn.py rel_logits_1drs77LAq!S U__R $ $A "aQ"A EE!aV  $ $Q 'E EE%!QU $E MM"a!eQUQY /E a!QUVmA !Q1a ''B2r:A 99\ ""c(eZdZdZfdZdZxZS) PosEmbedRelz Relative Position Embedding As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 cdt|t|\|_|_||_t jtj|jdzdz ||z|_ t jtj|jdzdz ||z|_ y)Nr r) super__init__rheightwidthdim_headnn Parametertorchrandn height_rel width_rel)self feat_sizer(scale __class__s rr%zPosEmbedRel.__init__=s "+I"6 TZ  ,,u{{4;;?Q3F'QTY'YZekk$**q.12Dh&ORW&WXr c*|j\}}}|j||j|jd}t ||j d}|j dd}t ||jd}||z}|j|||}|S)Nr )rrr )r rr )rr4rr5r )rrr&r'rr.rr-)r/rrHW_ rel_logits_w rel_logits_h rel_logitss rforwardzPosEmbedRel.forwardDs772q IIadjj" 5$Q_U  KK1 $QoV !L0 ''2r2 r )__name__ __module__ __qualname____doc__r%r; __classcell__r2s@rr"r"8sY r r"c4eZdZdZ dfd ZdZdZxZS)BottleneckAttna Bottleneck Attention Paper: `Bottleneck Transformers for Visual Recognition` - https://arxiv.org/abs/2101.11605 The internal dimensions of the attention module are controlled by the interaction of several arguments. * the output dimension of the module is specified by dim_out, which falls back to input dim if not set * the value (v) dimension is set to dim_out // num_heads, the v projection determines the output dim * the query and key (qk) dimensions are determined by * num_heads * dim_head if dim_head is not None * num_heads * (dim_out * attn_ratio // num_heads) if dim_head is None * as seen above, attn_ratio determines the ratio of q and k relative to the output if dim_head not used Args: dim (int): input dimension to the module dim_out (int): output dimension of the module, same as dim if not set stride (int): output stride of the module, avg pool used if stride == 2 (default: 1). num_heads (int): parallel attention heads (default: 4) dim_head (int): dimension of query and key heads, calculated from dim_out * attn_ratio // num_heads if not set qk_ratio (float): ratio of q and k dimensions to output dimension when dim_head not set. (default: 1.0) qkv_bias (bool): add bias to q, k, and v projections scale_pos_embed (bool): scale the position embedding as well as Q @ K c t ||Jd|xs|}||zdk(sJ||_|xst||zd|z|_||jz|_||jz|_||j z|_|jdz|_| |_ tj||j dz|jzd||_ t||j|j |_|dk(rtjddntj |_|j%y) NzBA concrete feature size matching expected input (H, W) is requiredr)divisorr r)bias)r(r1)r$r% num_headsr dim_head_qk dim_head_v dim_out_qk dim_out_vr1scale_pos_embedr)Conv2dqkvr" pos_embed AvgPool2dIdentitypoolreset_parameters) r/rdim_outr0striderIr(qk_ratioqkv_biasrNr2s rr%zBottleneckAttn.__init__js% $j&jj$.S"a'''"#a~g6HRS'TXa'a!T^^3#d&6&66"T__4%%- .99S$//A"5"FPXY%Y9I9IQUQ[Q[\*0A+BLLA&2;;=  r c:t|jj|jjjddzt|jj |j t|jj|j y)NrrG)std)rrPweightrrQr-r1r.)r/s rrUzBottleneckAttn.reset_parameterss\dhhoo488??+@+@+Ct+KLdnn//TZZ@dnn..DJJ?r c|j\}}}}t||jjk(dt||jjk(d|j |}t j||j|j|jgd\}}}|j||jz|jdjdd}|j||jz|jd}|j||jz|jdjdd}|jr%||z|j|z|j z} n$||z|j z|j|z} | j#d} | |zjddj||j||} |j%| } | S)Nr)rr r )rrrQr&r'rPr+splitrLrMrrIrJrrKrNr1softmaxrT) r/rrCrrrkvattnouts rr;zBottleneckAttn.forwardsWW 1aT^^***B/T^^)))2. HHQK++a$//4??DNN!SYZ[1a IIa$..($*:*:B ? I I"b Q IIa$..($*:*:B ? IIa$..($//2 > H HR P   EDNN1--;DETZZ'$..*;;D|||#ax""2r*221dnnaKiin r )NNrr5Ng?FF)r<r=r>r?r%rUr;r@rAs@rrCrCTs",VZ:? 0@ r rC)r?typingrr+torch.nnr)torch.nn.functional functionalrhelpersrr weight_initr trace_utilsrintrModuler"rCr rrpsV  .& #$s)#8"))8IRYYIr