kh]dZddlZddlZddlmZmZmZmZmZm Z ddl Z ddl m Z ddl m cm ZddlmZmZddlmZddlmZej,eZGd d e j2ZGd d eZ d"d eedededefdZe j@Z!e!fdeeefd eeefdedede jDde jFde jHfdZ%e!fde jHde jHdeeefde jFde jFde jHf dZ& d"de jHd eedededef dZ'Gdde j2Z(Gd d!e j2Z)y)#a? Image to Patch Embedding using Conv2d A convolution based approach to patchifying a 2D image w/ embedding projection. Based on code in: * https://github.com/google-research/vision_transformer * https://github.com/google-research/big_vision/tree/main/big_vision Hacked together by / Copyright 2020 Ross Wightman N)CallableDictListOptionalTupleUnion)nn)Formatnchw_to) to_2tuple)_assertc|eZdZUdZeed<ejje ed< dde e e e e ffde de de de ed e de ed e d e de ffd Zde e e e e fffd Z dde e e e e e ffde e e e e e fffdZdde e e e fe ffdZde e e fde e e ffdZdZxZS) PatchEmbed! 2D Image to Patch Embedding output_fmtdynamic_img_padimg_size patch_sizein_chans embed_dim norm_layerflattenbiasstrict_img_sizec t |t||_|j |\|_|_|_|d|_t||_ n||_tj|_ | |_ | |_ tj||||||_|r|||_ytj"|_y)NF kernel_sizestrider)super__init__r r_init_img_sizer grid_size num_patchesrr rNCHWrrr Conv2dprojIdentitynorm) selfrrrrrrrrrr __class__s S/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/layers/patch_embed.pyr!zPatchEmbed.__init__ s #J/:>:M:Mh:W7 t~t'7  ! DL$Z0DO#DL$kkDO..IIh zR\cgh -7Jy) R[[] c|jsJ|yt|}tt||jDcgc] \}}||z c}}}|d|dz}|||fScc}}w)N)NNNrr )rr tuplezip)r*rspr#r$s r,r"zPatchEmbed._init_img_size>sn  #X&c(DOO.LMda16MN lYq\1 K//NsA" c d}| t|}|||jk7rtj5t j |j j|j j|||j jdu}|jjt|j j|d|j j/|jj|j j||_ddd||_|xs |j}||jk7s|%|j|\|_ |_|_yy#1swYWxYw)NrT)verbose)r rtorchno_gradr r&r' in_channels out_channelsrweightcopy_resample_patch_embedrr"r#r$)r*rrnew_patch_sizenew_projs r,set_input_sizezPatchEmbed.set_input_sizeGs+   !&z2N  %.DOO*K %99II))II** .)t3 %%&:499;K;K^ei&jk99>>-MM'' 7$  %-DO,t}} t}} $(B>B>Q>QRZ>[ ;DM4>4+;)C % %s C E--E6returncH|rt|jS|jSN)maxr)r* as_scalars r, feat_ratiozPatchEmbed.feat_ratioas t' '?? "r-c|jrPtj|d|jdz tj|d|jdz fS|d|jdz|d|jdzfS)z Get grid (feature) size for given image size taking account of dynamic padding. NOTE: must be torchscript compatible so using fixed tuple indexing rr )rmathceilr)r*rs r,dynamic_feat_sizezPatchEmbed.dynamic_feat_sizegs   99Xa[4??1+==> (ST+X\XgXghiXjJj@kk kA;$//!"44hqkT__UVEW6WW Wr-c|j\}}}}|j|jrat||jdk(d|d|jddt||jdk(d|d|jddnr|jsft||j dzdk(d|d|j ddt||j dzdk(d|d|j dd|jr~|j d||j dzz |j dz}|j d||j dzz |j dz}t j|d|d|f}|j|}|jr"|jdjdd}n3|jtjk7rt||j}|j|}|S) NrzInput height (z) doesn't match model ().r z Input width (z%) should be divisible by patch size ()shaperrrrrFpadr'r transposerr r%r r))r*xBCHWpad_hpad_ws r,forwardzPatchEmbed.forwardpsWW 1a == $##T]]1--sBYZ^ZgZghiZjYkkm/noT]]1--qcAXY]YfYfghYiXjjl/mn))**a/$QC'LT__]^M_L``bc**a/#A3&KDOO\]L^K__ab   __Q'!dooa.@*@@DOOTUDVVE__Q'!dooa.@*@@DOOTUDVVEa!UAu-.A IIaL << ! &&q!,A __ +4??+A IIaLr-) NTNTTF)NN)T)__name__ __module__ __qualname____doc__r __annotations__r5jitFinalboolrintrrrstrr!r"r>rDrHrW __classcell__r+s@r,rrsYY__T**58  -1 (,$($)KCsCx01KK K  K !* KK! KK"K"K<0uS%S/-A'B0?C@D\uS%S/%9:;\!sE#s(O';!<=\4#E%S/32F,G# X%S/XeCHoXr-rceZdZUdZeed< d deededededeede dee d e ffd Z d e e jeeffd ZxZS)PatchEmbedWithSizerrrrrrrrrc 4t |||||||||y)N)rrrrrrrr)r r!) r*rrrrrrrrr+s r,r!zPatchEmbedWithSize.__init__s/ !!!  r-r?c4|j\}}}}|jft||jdzdk(d|d|jddt||jdzdk(d|d|jdd|j |}|jdd}|j r"|j dj dd}n3|jtjk7rt||j}|j|}||fS) NrzInput image height (z#) must be divisible by patch size (rJr zInput image width (rK) rLrrrr'rrOrr r%r r))r*rPrQrRrSrT feat_sizes r,rWzPatchEmbedWithSize.forwards+WW 1a == $ A**a/3GsJmnrn}n}~oAnBBD2E F A**a/3FqcIlmqm|m|}~mmAAC2D E IIaLGGBCL << ! &&q!,A __ +4??+A IIaL)|r-)rXrYrZr[NTNT)r\r]r^r_r r`rrdrrcrer!rr5TensorrrWrfrgs@r,riris'*  -1 (, sm      !*   !   , E%,,S "9: r-rinew_size interpolation antialiasr4c  ddl ddlm}t |j dk(sJdt dk(sJd|j dd}t|tk(r|S|r)tjd |j d d d fd  fd}||}tj jj|j|jfd} ||| dddd} |j} |j!}| |}|j#| }|S#t$r ddlm}YAwxYw)a/Resample the weights of the patch embedding kernel to target resolution. We resample the patch embedding kernel by approximately inverting the effect of patch resizing. Code based on: https://github.com/google-research/big_vision/blob/b00544b81f8694488d5f36295aeb7972f3755ffe/big_vision/models/proj/flexi/vit.py With this resizing, we can for example load a B/8 filter into a B/16 model and, on 2x larger input image, the result will match. Args: patch_embed: original parameter to be resized. new_size (tuple(int, int): target shape (height, width)-only. interpolation (str): interpolation for resize antialias (bool): use anti-aliasing filter in resize verbose (bool): log operation Returns: Resized patch embedding kernel. rN)vmapzFour dimensions expectedrKzNew shape should only be hwrlzResize patch embedding z to z, w/ z interpolation.ctj|d}tj||dj }|S)N)NN.)sizemoderq)rr.)r5rnrM interpolatenumpy)x_np _new_sizex_tf x_upsampledrqrps r,resizez(resample_patch_embed_old..resizesI||D!/2mm y} KKTVV[V[V] r-cg}tj|D]O}j|}d|j||<|j ||j dQj |jS)Ng?)rangeprodzeros unravel_indexappendreshapestackT) _old_sizer{mati basis_vecnpr~s r,get_resize_matz0resample_patch_embed_old..get_resize_mats~rwwy)* AA+I8:Ib&&q)4 5 JJvi3;;B? @ Axx}r-)devicecN|jdz}|jS)Nr)r)kernelresampled_kernelroresize_mat_pinvs r,resample_kernelz1resample_patch_embed_old..resample_kernels(*V^^B-??''11r-r )ryr5rs ImportError functorchlenrLr/_loggerinfotensorlinalgpinvrrdtypefloatto) patch_embedrorprqr4rsold_sizer resize_matrv_resample_kernel orig_dtyperr~rs ``` @@@r,resample_patch_embed_oldrsM4# {  !Q &B(BB & x=A <<<   %H X%/) .{/@/@.AhZuUbTccrst  (3Jll299>>*,,#? HZHZ[O2T/1a8!Q?""J##%K#K0K..,K M #""#sEEErrrr?c|\}}|\}} ||z} || z} tj| ||} | j| d||} tj| |||d}|j dj dddj| | }|S)zKComputes the resize matrix basis vectors and interpolates them to new_size.)rrr F)rvrwrq align_cornersrKr)r5eyerrMrxsqueezepermute)rrorprqrrold_hold_wnew_hnew_w old_total new_total eye_matrixbasis_vectors_batchresized_basis_vectors_batch resize_matrixs r,_compute_resize_matrixrsLE5LE5 I I9V5AJ$,,Y5%H"#--   #077:BB1aKSST]_hiM r-r pinv_matrixnew_size_tuplerintermediate_dtypec|j^}}}|j||dj|}|j|}||z}|j||g|j|}|S)zW Simplified resampling w/o vmap use. As proposed by https://github.com/stas-sl r)r)rLrr) rrrrrc_outc_in_resampled_patch_embeds r,_apply_resamplingrs"''OE4!%%eT2699@R9SK..'9.:K'+59199%WWZZakZl  r-c~t|jdk(sJdt|dk(sJdt|jdd}t|}||k(r|S|j}|j}t |||||t } tjj| } t|| ||t } | S)z5 Standalone function (computes matrix on each call). rtz/Input tensor should be 4D (out_ch, in_ch, h, w)rKz+New shape should only be hw (height, width)rlN) rrLr/rrrDTYPE_INTERMEDIATEr5rrr) rrorprqr4old_size_tuplerrrrrrs r,r;r;.s {  !Q &Y(YY & x=A LLL &+K,=,=bc,B&CN&+HoN'   F""J' y&J\J,,##J/K-[.*>P ! r-c eZdZdZ d deeefdedeffd Ze fdeeefde jde jd e jfd Zd e jdeed e jfd ZxZS) PatchEmbedResamplerFixedOrigSizez Resample patch embedding weights from a fixed original size, caching the pseudoinverse matrix based on the target size. orig_sizerprqct|t|trt |dk(sJd||_||_||_i|_y)z Args: orig_size (Tuple[int, int]): The expected original (height, width) of input patch_embed tensors. interpolation (str): Interpolation mode. antialias (bool): Use anti-aliasing filter in resize. rKz.`orig_size` must be a tuple of (height, width)N) r r! isinstancer/rrrprq_pinv_cache_map)r*rrprqr+s r,r!z)PatchEmbedResamplerFixedOrigSize.__init__QsS )U+I!0C = < =C"*";=r-rorrr?c|}|jj|}|r8t||r,t||}|j|k(r|j |k(r|St |j||j|j||}tjj|}d|dd|d}t||r t|||j||||j|<|S)zRRetrieves the cached pinv matrix or computes and caches it for the given new_size.pinv_rrPr )rgethasattrgetattrrrrrrprqr5rrdelattrregister_buffer)r*rorr cache_key buffer_namerrs r,_get_or_create_pinv_matrixz;PatchEmbedResamplerFixedOrigSize._get_or_create_pinv_matrixfs **..y9 745!$ 4K!!V+ 0A0AU0J##, NNHd&8&8$..&RW ll'' 3 hqk]!HQK=9 4 % T; ' [+6*5Y'r-rc~t|jdk(sJt|dk(sJt|jdd}||jk(sJd|d|jt|}|j|k(r|S|j}|j }|j ||}t||||}|S)a Resamples the patch embedding weights to new_size. Args: patch_embed (torch.Tensor): Original weights (out_ch, in_ch, H_orig, W_orig). new_size (List[int]): Target [height, width]. Returns: torch.Tensor: Resampled weights. rtrKrlNzInput patch_embed spatial size z0 does not match module's expected original size )rrLr/rrrrr) r*rro input_sizerrrrrs r,rWz(PatchEmbedResamplerFixedOrigSize.forwards;$$%***8}!!!;,,RS12 T^^+ @-j\://3~~.> @ @++0/ >>^ + ## && 55nfM !2+{N\f g$$r-)bicubicT)r\r]r^r_rrdrercr!rr5rrrnrrrWrfrgs@r,rrLs' >c?>> >20 S/ {{   <"%5<<"%49"%"%r-rc feZdZdZ ddeeefdedededef fd Zde jd eeefd e jfd Z de jd eeefd e jfd Z dd e jde jde e jde eeefded e jf dZxZS)PatchEmbedInterpolatora8Dynamically interpolates patch embedding weights for variable patch sizes. This module wraps patch embedding weight resampling functionality to support on-the-fly patch size variation during training. It handles both Conv2d and Linear patch embeddings. Args: base_patch_size: The original patch size the model was initialized with in_chans: Number of input channels embed_dim: Embedding dimension interpolation: Interpolation mode for resampling antialias: Whether to use antialiasing during interpolation base_patch_sizerrrprqcht|||_||_||_||_||_yrA)r r!rrrrprq)r*rrrrprqr+s r,r!zPatchEmbedInterpolator.__init__s5 .  "*"r-r9target_patch_sizer?cj||jk(r|S|jd}|j\}}|\}}|j||||j}|j dddd}t |||g|j |jd} | j dddd} | j|d} | S)a/Resample linear patch embedding weights for a new patch size. Args: weight: Linear weight tensor of shape [embed_dim, patch_h * patch_w * in_chans] target_patch_size: Target (patch_h, patch_w) to resample to Returns: Resampled weight tensor rrZr rKFrorprqr4r)rrLrrrr;rprq) r*r9rrbase_phbase_pw target_ph target_pw weight_convweight_conv_resampledweight_resampleds r,resample_linear_weightz-PatchEmbedInterpolator.resample_linear_weights  4 4 4MLLO //0 9nnY$--P !))!Q15 !5 +,,nn ! 188Aq!D+33IrBr-c||jk(r|St|t||j|jd}|S)a-Resample conv2d patch embedding weights for a new patch size. Args: weight: Conv2d weight tensor of shape [embed_dim, in_chans, patch_h, patch_w] target_patch_size: Target (patch_h, patch_w) to resample to Returns: Resampled weight tensor Fr)rr;listrprq)r*r9rrs r,resample_conv_weightz+PatchEmbedInterpolator.resample_conv_weightsK  4 4 4M0 +,,,nn   r-patches proj_weight proj_biasr is_linearc| |j}|r||jk7rz|jdk(sJd|j\}}}} } |j||} |j ||d} t j jj| | |} | S|jdk(r%|j\}}}} } |j ||d}t j jj|||} | S||jk7rB|j||} t j jj|| ||d} | St j jj||||d} | S)aWApply patch embedding with dynamic weight resampling. Args: patches: Input patches - For linear mode with resampling: [B, N, Ph, Pw, C] - For linear mode without resampling: [B, N, Ph*Pw*C] - For conv mode: [B, C, H, W] proj_weight: Original projection weight proj_bias: Optional projection bias patch_size: Current patch size (if None, uses base_patch_size) is_linear: Whether using linear (True) or conv2d (False) projection Returns: Embedded patches z0Patches must be [B, N, Ph, Pw, C] for resamplingrr)rpadding) rndimrLrrr5r functionallinearrconv2d)r*rrrrrrQNPhPwrRr patches_flatoutputs r,rWzPatchEmbedInterpolator.forward s.  --J T111||q(\*\\(")--1b"a$(#>#>{J#W  'q!R8 ,,33LBRT]^* %<<1$&-mmOAq"b!%ooaB7G,,33G[)T T111#'#<#<[*#U ,,33-y%q4 ,,33[)%q4  r-)rZr[rT)NNT)r\r]r^r_rrdrercr!r5rnrrrrWrfrgs@r,rrs# "& #sCx # # #  #  #(  ( !c?(   ( T   !c?    @-104 ::\\:ELL) : U38_- :  : :r-r)rTF)*r_loggingrFtypingrrrrrrr5r torch.nn.functionalrrMformatr r helpersr trace_utilsr getLoggerr\rModulerrirdrercrfloat32rrrrnrrr;rrr-r,rs  ?? # '  H %ood((^' Cs)CC C  CL]], CHoCHo  LL  ;;  \\@'9 !!!#s(O! !  !  \\ !*' !\\!s)!! !  !