kh`}ddlZddlmZddlmZddlmZmZmZm Z m Z m Z ddl Z ddlZddlmcmZddlmZmZddlmZmZmZddlmZddlmZmZdd lmZmZdd l m!Z!dd l"m#Z#m$Z$dd l%m&Z&gd Z'de e(e(fde(de(de(de e(e(ff dZ)de e(e(fde(dee e(e(ffdZ*de(de(dej&fdZ+GddejXZ-GddejXZ.GddejXZ/Gdd ejXZ0Gd!d"ejXZ1Gd#d$ejXZ2Gd%d&ejXZ3Gd'd(ejXZ4Gd)d*ejXZ5 d2;-S"))^,;-$;- ;-z  F r'rDcdeZdZdZdedededdffd ZdejfdZd edefd Z xZ S) $RelativePositionalMultiHeadAttentionzRelative Positional Multi-Head Attention. Args: feat_dim (int): Number of input features. head_dim (int): Number of features per head. max_seq_len (int): Maximum sequence length. feat_dimhead_dim max_seq_lenr Ncbt|||zdk7rtd|d|||z|_||_t t j||_||_ tj||j|jzdz|_ |dz|_ tj|j|jz||_tjj!t#j$d|jzdz d|jzdz z|jft"j&|_|j+d t-|j|jt"jj.j1|j(d y) Nrz feat_dim: z must be divisible by head_dim: r*gr"r#)dtyperelative_position_index{Gz?std)rZr[ ValueErrorn_headsr|ramathsqrtsizer}r Linearto_qkv scale_factormerge parameter Parameterr7emptyfloat32relative_position_bias_tableregister_bufferrBinit trunc_normal_)rer{r|r}rjs r%r[z-RelativePositionalMultiHeadAttention.__init__sU  h ! #z(3ST\S]^_ _8+    +./ &ii$,,*F*JK $dNYYt}}t||;XF ,.LL,B,B KK!dii-!+DII 0ABDLLQY^YfYf g- ) 68TUYU^U^`d`i`i8jk  ##D$E$E4#Pr'c|jjd}|j|j|j|jd}|j dddj }|j dS)Nr6r"rr#)rviewrr}r<r= unsqueeze)re bias_index relative_biass r%get_relative_positional_biaszARelativePositionalMultiHeadAttention.get_relative_positional_biasss1166r: 99*EJJ4K[K[]a]m]moqr %--aA6AAC &&q))r'rkc|j\}}}}|j|j}}|j|}t j |dd\} } } | j |||||jddddd} | j |||||jddddd} | j |||||jddddd} | |jz} t jd| | } |j} tj| | zd} t jd | | }|jdddddj ||||}|j|}|S) z Args: x (Tensor): Input tensor with expected layout of [B, G, P, D]. Returns: Tensor: Output tensor with expected layout of [B, G, P, D]. r*r6)dimrr#r"z!B G H I D, B G H J D -> B G H I Jz!B G H I J, B G H J D -> B G H I D)shaperr|rr7chunkreshaper<reinsumrFsoftmaxr)rerkBGPDHDHqkvqkvdot_prodpos_biasouts r%roz,RelativePositionalMultiHeadAttention.forwardsXWW 1a dmm2kk!n++c1"-1a IIaAq" % - -aAq! < IIaAq" % - -aAq! < IIaAq" % - -aAq! < !! !<< CQJ44699X0b9ll>!Lkk!Q1a(00Aq!<jjo r') rqrrrsrtrar[r7r rrorwrxs@r%rzrzs[QQQ Q  Q8*ell* Fr'rzcheZdZdZdededdffd ZdejdejfdZxZ S) SwapAxeszPermute the axes of a tensor.abr Nc>t|||_||_yN)rZr[rr)rerrrjs r%r[zSwapAxes.__init__s r'rkc\tj||j|j}|Sr)r7swapaxesrrrms r%rozSwapAxes.forwards!nnQ/ r') rqrrrsrtrar[r7r rorwrxs@r%rrs;'##$ %,,r'rc8eZdZdZdfd ZdededefdZxZS)WindowPartitionzB Partition the input tensor into non-overlapping windows. r c"t|yrrZr[rerjs r%r[zWindowPartition.__init__ r'rkpc|j\}}}}|}|j||||z|||z|}|jdddddd}|j|||z||zz||z|}|S)z Args: x (Tensor): Input tensor with expected layout of [B, C, H, W]. p (int): Number of partitions. Returns: Tensor: Output tensor with expected layout of [B, H/P, W/P, P*P, C]. rr"rr*r#rrr<)rerkrrCrWrs r%rozWindowPartition.forwards|WW 1a  IIaAFAqAvq 1 IIaAq!Q ' IIa!q&Q!V,a!eQ 7r'r N rqrrrsrtr[r rarorwrxs@r%rrs'CFr'rc @eZdZdZd fd Zdededededef dZxZS) WindowDepartitionzo Departition the input tensor of non-overlapping windows into a feature volume of layout [B, C, H, W]. r c"t|yrrrs r%r[zWindowDepartition.__init__rr'rkr h_partitions w_partitionsc|j\}}}}|} ||} } |j|| | | | |}|jdddddd}|j||| | z| | z}|S)ar Args: x (Tensor): Input tensor with expected layout of [B, (H/P * W/P), P*P, C]. p (int): Number of partitions. h_partitions (int): Number of vertical partitions. w_partitions (int): Number of horizontal partitions. Returns: Tensor: Output tensor with expected layout of [B, C, H, W]. rrr#r*r"rr) rerkrrrrrPPrrHPWPs r%rozWindowDepartition.forwardstgg 1b! |B IIaRAq ) IIaAq!Q ' IIaBFBF +r'rrrxs@r%rrs6Cs#RXr'rceZdZdZdededededeeefdeded ejfd ed ejfd e d e d e ddffd Z de de fdZ xZS)PartitionAttentionLayera Layer for partitioning the input tensor into non-overlapping windows and applying attention to each window. Args: in_channels (int): Number of input channels. head_dim (int): Dimension of each attention head. partition_size (int): Size of the partitions. partition_type (str): Type of partitioning to use. Can be either "grid" or "window". grid_size (Tuple[int, int]): Size of the grid to partition the input tensor into. mlp_ratio (int): Ratio of the feature size expansion in the MLP layer. activation_layer (Callable[..., nn.Module]): Activation function to use. norm_layer (Callable[..., nn.Module]): Normalization function to use. attention_dropout (float): Dropout probability for the attention layer. mlp_dropout (float): Dropout probability for the MLP layer. p_stochastic_dropout (float): Probability of dropping out a partition. rEr|partition_sizepartition_type grid_size mlp_ratiorI.rJattention_dropout mlp_dropoutrKr Nc t |||z|_||_|d|z|_||_||_|dvr td|dk(r||jc|_|_ n|j|c|_|_ t|_ t|_ |dk(r tddntj |_|dk(r tddntj |_tj&||t)|||dztj*| |_tj&tj.|tj0|||z|tj0||z|tj*| |_t5| d |_y) Nr)gridwindowz0partition_type must be either 'grid' or 'window'rrr"rOrP)rZr[rr| n_partitionsrrrrgr partition_oprdepartition_oprr r`partition_swapdepartition_swapr^rzDropout attn_layer LayerNormr mlp_layerrstochastic_dropout) rerEr|rrrrrIrJrrrKrjs r%r[z PartitionAttentionLayer.__init__,s" "h.   %aLN:," !3 3OP P X %+T->->NDFDF!..NDFDF+-/12@F2Jhr2.PRP[P[P]4Bf4LR 0RTR]R]R_-- { # 1hXYHY Z JJ( )   LL % IIk;#: ;   IIkI-{ ; JJ{ #  #22FU"Sr'rkc|jd|jz|jd|jz}}tj|jd|jzdk(xr|jd|jzdk(dj |j|j|j ||j}|j |}||j|j|z}||j|j|z}|j|}|j||j||}|S)z Args: x (Tensor): Input tensor with expected layout of [B, C, H, W]. Returns: Tensor: Output tensor with expected layout of [B, C, H, W]. rr#z[Grid size must be divisible by partition size. Got grid size of {} and partition size of {}) rrr7_assertformatrrrrrrr)rerkghgws r%rozPartitionAttentionLayer.forwardfs!"dff,dnnQ.?466.IB NN1  &! + Oq0ADFF0Ja0O i p p     a (    " ''(:; ; ''q(9: :  ! !! $   4662r 2r')rqrrrsrtrastrr rr rvrur[r rorwrxs@r%rrs"8T8T8T  8T  8Tc?8T8T#3 >28TS"))^,8T!8T8T$8T !8TtFr'rceZdZdZdededededededejfd edejfd ed ed ed ededede eefddffd Z de de fdZ xZ S) MaxVitLayera MaxVit layer consisting of a MBConv layer followed by a PartitionAttentionLayer with `window` and a PartitionAttentionLayer with `grid`. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. expansion_ratio (float): Expansion ratio in the bottleneck. squeeze_ratio (float): Squeeze ratio in the SE Layer. stride (int): Stride of the depthwise convolution. activation_layer (Callable[..., nn.Module]): Activation function. norm_layer (Callable[..., nn.Module]): Normalization function. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Ratio of the MLP layer. mlp_dropout (float): Dropout probability for the MLP layer. attention_dropout (float): Dropout probability for the attention layer. p_stochastic_dropout (float): Probability of stochastic depth. partition_size (int): Size of the partitions. grid_size (Tuple[int, int]): Size of the input feature grid. rErFrHrGrrJ.rIr|rrrrKrrr Nc 2t|t}t|||||||| |d<t ||| d|| |t j | | |  |d<t ||| d|| |t j | | |  |d<t j||_y)N)rErFrGrHrrIrJrKMBconvr) rEr|rrrrrIrJrrrKwindow_attentionrgrid_attention) rZr[rrDrr rr^rd)rerErFrHrGrrJrIr|rrrrKrrrdrjs r%r[zMaxVitLayer.__init__s* )m"#%+'-!!5  x&=$)#-||/#!5 & !"$;$)!-||/#!5 $  mmF+ r'rkc(|j|}|Sz Args: x (Tensor): Input tensor of shape (B, C, H, W). Returns: Tensor: Output tensor of shape (B, C, H, W). rd)rerks r%rozMaxVitLayer.forwards KKNr')rqrrrsrtrarurr rvr r[r rorwrxs@r%rrs(?,?, ?,  ?,  ?,?,S"))^,?,#3 >2?,?,?,?,!?, $!?,$%?,&c?'?,( )?,BFr'rceZdZdZdedededededejfdedejfd ed ed ed ed ede eefdede eddffd Z de de fdZ xZS) MaxVitBlocka( A MaxVit block consisting of `n_layers` MaxVit layers. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. expansion_ratio (float): Expansion ratio in the bottleneck. squeeze_ratio (float): Squeeze ratio in the SE Layer. activation_layer (Callable[..., nn.Module]): Activation function. norm_layer (Callable[..., nn.Module]): Normalization function. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Ratio of the MLP layer. mlp_dropout (float): Dropout probability for the MLP layer. attention_dropout (float): Dropout probability for the attention layer. p_stochastic_dropout (float): Probability of stochastic depth. partition_size (int): Size of the partitions. input_grid_size (Tuple[int, int]): Size of the input feature grid. n_layers (int): Number of layers in the block. p_stochastic (List[float]): List of probabilities for stochastic depth for each layer. rErFrHrGrJ.rIr|rrrrinput_grid_sizen_layers p_stochasticr Ncpt|t|| k(std| d|dt j |_t| ddd|_t|D]L\}}|dk(rdnd}|xj t|dk(r|n|||||||||| | | |j| gz c_Ny) Nz'p_stochastic must have length n_layers=z, got p_stochastic=.r*r"r#rNr)rErFrHrGrrJrIr|rrrrrrK) rZr[lenrr ModuleListrdr&r enumerater)rerErFrHrGrJrIr|rrrrrrridxrrrjs r%r[zMaxVitBlock.__init__s, < H,FxjPcdpcqqrst tmmo /QWXbcd - FC(QF KK/2ax \!-"/$3!)%5%' +&7#1"nn)* K r'rkc8|jD] }||} |Srr)rerklayers r%rozMaxVitBlock.forward,s%[[ EaA r')rqrrrsrtrarurr rvr rr[r rorwrxs@r%rrs*11 1  1  1S"))^,1#3 >21111!1 !1"sCx#1&'1(5k)1* +1f  F r'rc!eZdZdZdej ddddddfdeeefded ed eed eed ed e de e dejfde dejfde de dede de deddf fd Z dedefdZdZxZS)ray Implements MaxVit Transformer from the `MaxViT: Multi-Axis Vision Transformer `_ paper. Args: input_size (Tuple[int, int]): Size of the input image. stem_channels (int): Number of channels in the stem. partition_size (int): Size of the partitions. block_channels (List[int]): Number of channels in each block. block_layers (List[int]): Number of layers in each block. stochastic_depth_prob (float): Probability of stochastic depth. Expands to a list of probabilities for each layer that scales linearly to the specified value. squeeze_ratio (float): Squeeze ratio in the SE Layer. Default: 0.25. expansion_ratio (float): Expansion ratio in the MBConv bottleneck. Default: 4. norm_layer (Callable[..., nn.Module]): Normalization function. Default: None (setting to None will produce a `BatchNorm2d(eps=1e-3, momentum=0.01)`). activation_layer (Callable[..., nn.Module]): Activation function Default: nn.GELU. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Expansion ratio of the MLP layer. Default: 4. mlp_dropout (float): Dropout probability for the MLP layer. Default: 0.0. attention_dropout (float): Dropout probability for the attention layer. Default: 0.0. num_classes (int): Number of classes. Default: 1000. Ng?rrpir stem_channelsrblock_channels block_layersr|stochastic_depth_probrJ.rIrHrGrrr num_classesr ct|t|d}|ttj dd}t |t|}t|D]3\}}|d|zdk7s |d|zdk7std|d|d |d |d t jt||dd || d dt||ddddd|_ t|dd d}||_t j|_|g|ddz}|}t#j$d|t'|j)}d}t+|||D]\\}}}|j j-t/||| | || || | ||||||||z|j dj0}||z }^t jt j2dt j4t j6|dt j8|d|dt j:t j8|d|d |_|j?y)Nr*gMbP?g{Gz?)epsmomentumrr#z Input size z of block z$ is not divisible by partition size zx. Consider changing the partition size or the input size. Current configuration yields the following block input sizes: rr"F)rrJrIrMrST)rrJrIrMrNr6)rErFrHrGrJrIr|rrrrrrr)rM) rZr[rrr BatchNorm2dr0rrrr^rstemr&rrblocksnplinspacer>tolistzipr,rrAdaptiveAvgPool2dFlattenrrTanh classifier _init_weights)rerrrrrr|rrJrIrHrGrrrr input_channelsblock_input_sizesrblock_input_sizerErFrp_idx in_channel out_channel num_layersrjs r%r[zMaxVit.__init__Ms: D!   TDIJ 5Z^ATU%./@%A  !C!"^3q8 IIaL[[ EaA  OOA r'c|jD]l}t|tjrbtjj |j d|jVtjj|jt|tjrUtjj|j dtjj|jdt|tjs tjj |j d|jDtjj|joy)Nrrr#r) modules isinstancer r\rnormal_weightrMzeros_r  constant_r)rems r%rzMaxVit._init_weightss +A!RYY'd366%GGNN166*Ar~~.!!!((A.!!!&&!,Aryy)d366%GGNN166* +r')rqrrrsrtr GELUr rarrurrrvr[r rorrwrxs@r%rr8s%J:>57WW#!" #&7t#s(Ot  ttS t3itt %t"Xc299n56#t$#3 >2%t()t*+t./t01t2!3t67t8 9tlF +r'rrrrrrr|weightsprogresskwargsc d|dt|dt|jd|jdd|jddk(sJt|d|jd|jdd} t d ||||||| d|} |"| j |j |d | S) Nr  categoriesmin_sizerr#rr2)rrrrr|rrT)r, check_hashr$)rrmetapoprload_state_dictget_state_dict) rrrrrr|r+r,r-rmodels r%_maxvitr9s$fmSl9S5TU||J'*gll:.Fq.IIIIflGLL4LML*5J  #%!3%   E g44hSW4XY Lr'c jeZdZedeeddejedddddd d id d d dZ e Z y)rz9https://download.pytorch.org/models/maxvit_t-bc5ab103.pthr2) crop_size resize_size interpolationir1zLhttps://github.com/pytorch/vision/tree/main/references/classification#maxvitz ImageNet-1KgT@g|?5.X@)zacc@1zacc@5gZd;@gK7]@zThese weights reproduce closely the results of the paper using a similar training recipe. They were trained with a BatchNorm2D momentum of 0.99 instead of the more correct 0.01.)r/ num_paramsr0recipe_metrics_ops _file_size_docs)url transformsr4N) rqrrrsrrrrBICUBICr IMAGENET1K_V1DEFAULTr$r'r%rrsb G 3CO`OhOh /""d##  !g M.Gr'r pretrained)r+T)r+r,c \tj|}tddgdgdddd||d|S) a Constructs a maxvit_t architecture from `MaxViT: Multi-Axis Vision Transformer `_. Args: weights (:class:`~torchvision.models.MaxVit_T_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.MaxVit_T_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.maxvit.MaxVit`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.MaxVit_T_Weights :members: @)rKi)r"r"rr" g?)rrrr|rrr+r,r$)rverifyr9)r+r,r-s r%rrsH.%%g.G  *!!    r')NF)r`sX #AA HH9TH<R1 uS#XSRU`chmnqsvnvhwsCxCDQVWZ\_W_Q`La##S#U\\#TRYYTnF299FR ryy bii4