khMLdZddlmZmZmZmZddlZddlmZddlm Z ddl m Z ddl mZdd lmZmZmZmZmZmZdd lmZmZdd lmZ dd lmZd$d ZGddej>Z GddejBZ"d%dZ#GddejjHZ%dZ&dZ'dZ(GddejRZ*GddejRZ+GddejXZ-GddejXZ.Gd d!eZ/Gd"d#eZ0y#e$r dd lmZYwxYw)&aw Normalization + Activation Layers Provides Norm+Act fns for standard PyTorch norm layers such as * BatchNorm * GroupNorm * LayerNorm This allows swapping with alternative layers that are natively both norm + act such as * EvoNorm (evo_norm.py) * FilterResponseNorm (filter_response_norm.py) * InplaceABN (inplace_abn.py) Hacked together by / Copyright 2022 Ross Wightman )UnionListOptionalAnyN)nn) functional)FrozenBatchNorm2d)create_act_layer) is_fast_normfast_group_normfast_layer_norm fast_rms_norm rms_norm2dfast_rms_norm2d)RmsNorm RmsNorm2d)_assert)rms_normc|xsi}|jd|d}|r t|fi|}|tjS|S)Ninplace) setdefaultr rIdentity) act_layer act_kwargsr apply_actacts P/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/layers/norm_act.py _create_actr!sH!rJ)W- Cy7J7K2;;=0S0c TeZdZdZdddddej dddddf fd ZdZxZS)BatchNormAct2dzBatchNorm + Activation This module performs BatchNorm + Activation in a manner that will remain backwards compatible with weights trained with separate bn, act. This is why we inherit from BN instead of composing it as a .bn member. h㈵>g?TNc  | | d} tt| |f||||d| | | nt j |_t||| ||_y#t$rtt| |||||YZwxYw)N)devicedtype)epsmomentumaffinetrack_running_statsrrr) superr"__init__ TypeErrorrrdroprr)self num_featuresr'r(r)r*rrrr drop_layerr%r&factory_kwargs __class__s rr-zBatchNormAct2d.__init__1s (.?N .$ 0 !$7   !  "%/$:JL  yZ\ef  .$ 0!$7 1  sA!A<;A<c t|jdk(d|jd|jd}n |j}|jrd|jrX|j L|j j d|jdt|j z }n |j} |jrd}n|jduxr|jdu} tj||jr |jr |jnd|jr |jr |jnd|j|j|||j}|j|}|j!|}|S)Nzexpected 4D input (got zD input)gr g?T)rndimr(trainingr*num_batches_trackedadd_float running_mean running_varF batch_normweightbiasr'r/r)r0xexponential_average_factor bn_trainings rforwardzBatchNormAct2d.forwardVsR! 6qvvhhGH == ), &)- & ==T55''3((--a0==(14uT=U=U7V1V.15.  ==K,,4T4;K;Kt;SK LL %)]]d6N6ND  TX$(MMT5M5MD  SW KK II  & HH  IIaL HHQKr ) __name__ __module__ __qualname____doc__rReLUr-rE __classcell__r4s@rr"r"*s; $gg#gJ0r r"cPeZdZdejdejffd ZxZS)SyncBatchNormActrBreturnct||}t|dr|j|}t|dr|j |}|S)Nr/r)r,rEhasattrr/r)r0rBr4s rrEzSyncBatchNormAct.forwardsD GOA  4  ! A 4  Ar )rFrGrHtorchTensorrErKrLs@rrNrNs# %,,r rNc|}t|tjjjj rxt|t rft|j|j|j|j|j|}|j|_ |j|_nVtjj|j|j|j|j|j|}|jr?tj 5|j"|_|j$|_ddd|j&|_|j(|_|j*|_|j,|_t/|dr|j0|_|j3D]!\}}|j5|t7||#~|S#1swYxYw)N) process_groupqconfig) isinstancerRrmodules batchnorm _BatchNormr"rNr1r'r(r)r*rr/ SyncBatchNormno_gradr@rAr<r=r9r8rQrVnamed_children add_moduleconvert_sync_batchnorm)modulerU module_outputnamechilds rr_r_sM&%((**44??@ fn -,##  **+ M!' M !'M "HH22##  ** M == 1'-}} $%+[[ " 1&,%8%8 "$*$6$6 !,2,F,F )!'  69 %$*NNM !,,.U e  '=e]'STU  1 1s (#G++G4ceZdZdZddej dddfdedeffd Zde d e d e d e d e e d e e de e ffd Z dejdejfdZde fdZxZS)FrozenBatchNormAct2da$ BatchNormAct2d where the batch statistics and the affine parameters are fixed Args: num_features (int): Number of features ``C`` from an expected input of size ``(N, C, H, W)`` eps (float): a value added to the denominator for numerical stability. Default: 1e-5 r#TNr1r'ct|||_|jdt j ||jdt j ||jdt j ||jdt j |||ntj|_ t|||||_ y)Nr@rAr<r=r+) r,r-r'register_bufferrRoneszerosrrr/rr) r0r1r'rrrrr2r4s rr-zFrozenBatchNormAct2d.__init__s  Xuzz,'?@ VU[[%>? ^U[[-FG ]EJJ|,DE$.$:JL  yZ\efr state_dictprefixlocal_metadatastrict missing_keysunexpected_keys error_msgsc H|dz}||vr||=t ||||||||y)Nr9)r,_load_from_state_dict) r0rjrkrlrmrnrorpnum_batches_tracked_keyr4s rrrz*FrozenBatchNormAct2d._load_from_state_dicts?#)+@"@ "j 023 %  oWa r rBrOc|jjdddd}|jjdddd}|jjdddd}|jjdddd}|||j zj z}|||zz }||z|z}|j|j|}|S)Nr ) r@reshaperAr=r<r'rsqrtrr/)r0rBwbrvrmscalerAs rrEzFrozenBatchNormAct2d.forwards KK  2q! , II  aQ *    % %aQ 2    & &q"a 3R$((]))++2:~ I  HHTYYq\ "r c|jjd|jjdd|jd|j dS)N(rz, eps=z, act=))r4rFr@shaper'r)r0s r__repr__zFrozenBatchNormAct2d.__repr__sJ..))*!DKK,=,=a,@+AzQWX\X`X`Waabccr )rFrGrHrIrrJintr;r-dictstrboolrrrrRrSrErrKrLs@rreres''ggg(       3i  c I $  %,, d#dr recF|}t|ttfrDt|j}|j|_|j |_|j r|j jjj|j _|jjjj|j_|jj|j_|jj|j_|j|_ |j|_|j|_|St|t j"j$j&j(t j"j$j&j*fr"t-|j}|j|_|j |_|j r|j jjj|j _|jjjj|j_|jj|j_|jj|j_|j|_ |S|j/D]'\}}t1|}||us|j3||)|S)a Converts all `BatchNorm2d` and `SyncBatchNorm` or `BatchNormAct2d` and `SyncBatchNormAct2d` layers of provided module into `FrozenBatchNorm2d` or `FrozenBatchNormAct2d` respectively. Args: module (torch.nn.Module): Any PyTorch module. Returns: torch.nn.Module: Resulting module Inspired by https://github.com/pytorch/pytorch/blob/a5895f85be0f10212791145bfedc0261d364f103/torch/nn/modules/batchnorm.py#L762 )rWr"rNrer1r)r@dataclonedetachrAr<r=r'r/rrRrrXrY BatchNorm2dr[r r]freeze_batch_norm_2dr^r`resrbrc new_childs rrrs5 C&>+;<="6#6#67!..]] ==$mm00668??ACJJO"KK,,224;;=CHHM & 3 3 8 8%1166**;;** J FUXX--77CCUXXEUEUE_E_EmEmn o 3 34!..]] ==$mm00668??ACJJO"KK,,224;;=CHHM & 3 3 8 8%1166** J "002 0KD%,U3I%tY/ 0 Jr c.|}t|tr"t|j}|jr|j j jj|j _|jj jj|j_|jj |j_|jj |j_|j|_ |j|_ |j|_|St|trt j"j%|j}|jr|j j jj|j _|jj jj|j_|jj |j_|jj |j_|j|_ |S|j'D]'\}}t)|}||us|j+||)|S)a Converts all `FrozenBatchNorm2d` layers of provided module into `BatchNorm2d`. If `module` is itself and instance of `FrozenBatchNorm2d`, it is converted into `BatchNorm2d` and returned. Otherwise, the module is walked recursively and submodules are converted in place. Args: module (torch.nn.Module): Any PyTorch module. Returns: torch.nn.Module: Resulting module Inspired by https://github.com/pytorch/pytorch/blob/a5895f85be0f10212791145bfedc0261d364f103/torch/nn/modules/batchnorm.py#L762 )rWrer"r1r)r@rrrrAr<r=r'r/rr rRrrr]unfreeze_batch_norm_2dr^rs rrr.s C&./V001 ==$mm00668??ACJJO"KK,,224;;=CHHM & 3 3 8 8%1166**;;** J F- .hh""6#6#67 ==$mm00668??ACJJO"KK,,224;;=CHHM & 3 3 8 8%1166** J "002 0KD%.u5I%tY/ 0 Jr c(|r||zdk(sJ||zS|S)Nr) num_channels num_groups group_sizes r _num_groupsrWs)j(A---z)) r c LeZdZdddddejdddf fd ZdZxZS) GroupNormAct r#TNc tt| t||||||| | nt j |_t||| ||_t|_ y)Nr'r)r+) r,rr-rrrr/rrr _fast_norm) r0rrr'r)rrrrrr2r4s rr-zGroupNormAct.__init__`sb lD*  j* =  + %/$:JL  yZ\ef&.r cT|jr8t||j|j|j|j }nAt j||j|j|j|j }|j|}|j|}|SN rr rr@rAr'r> group_normr/rr0rBs rrEzGroupNormAct.forwardxp ??4??DKKDHHUA QdiiRA IIaL HHQKr rFrGrHrrJr-rErKrLs@rrr^s. gg)0r rcHeZdZdddejdddffd ZdZxZS) GroupNorm1Actr#TNc tt| d|||||ntj|_t |||||_t|_ y)Nr rr+) r,rr-rrr/rrr r r0rr'r)rrrrr2r4s rr-zGroupNorm1Act.__init__sO mT+A|V+T$.$:JL  yZ\ef&.r cT|jr8t||j|j|j|j }nAt j||j|j|j|j }|j|}|j|}|Srrrs rrEzGroupNorm1Act.forwardrr rrLs@rrrs(gg)"r rcteZdZdddejdddfdeeeeejfffd Z dZ xZ S) LayerNormActr#TNnormalization_shapec tt| |||||ntj|_t |||||_t|_ yN)r'elementwise_affiner+) r,rr-rrr/rrr r) r0rr'r)rrrrr2r4s rr-zLayerNormAct.__init__sO lD*+>C\b*c$.$:JL  yZ\ef&.r cT|jr8t||j|j|j|j }nAt j||j|j|j|j }|j|}|j|}|Sr) rrnormalized_shaper@rAr'r> layer_normr/rrs rrEzLayerNormAct.forwardsw ??4#8#8$++tyyRVRZRZ[A Q 5 5t{{DIItxxXA IIaL HHQKr ) rFrGrHrrJrrrrRSizer-rErKrLs@rrrsDgg)!&sDIuzz'A!B)"r rcHeZdZdddejdddffd ZdZxZS)LayerNormAct2dr#TNc tt| |||||ntj|_t |||||_t|_ yr) r,rr-rrr/rrr rrs rr-zLayerNormAct2d.__init__sN nd,\sW],^$.$:JL  yZ\ef&.r c|jdddd}|jr8t||j|j|j |j }nAtj||j|j|j |j }|jdddd}|j|}|j|}|S)Nrr ) permuterrrr@rAr'r>rr/rrs rrEzLayerNormAct2d.forwards IIaAq ! ??4#8#8$++tyyRVRZRZ[A Q 5 5t{{DIItxxXA IIaAq ! IIaL HHQKr rrLs@rrrs(gg) r rc~eZdZdZdddej dddffd ZdejdejfdZ xZ S) RmsNormAct* RMSNorm + Activation for '2D' NCHW tensors NOTE: It's currently (2025-05-10) faster to use an eager 2d kernel that does reduction on dim=1 than to permute and use internal PyTorch F.rms_norm, this may change if something like https://github.com/pytorch/pytorch/pull/150576 lands. ư>TNc t ||||||ntj|_t |||||_t|_yN)channelsr'r)r+ r,r-rrr/rrr rrs rr-zRmsNormAct.__init__K ,CG$.$:JL  yZ\ef&.r rBrOc|jr-t||j|j|j}n,t ||j|j|j}|j |}|j|}|Sr)rrrr@r'rr/rrs rrEzRmsNormAct.forwardsd ??a!6!6 TXXNAD114;;IA IIaL HHQKr rFrGrHrIrrJr-rRrSrErKrLs@rrrCgg) %,,r rc~eZdZdZdddej dddffd ZdejdejfdZ xZ S) RmsNormAct2drrTNc t ||||||ntj|_t |||||_t|_yrrrs rr-zRmsNormAct2d.__init__rr rBrOc|jr-t||j|j|j}n,t ||j|j|j}|j |}|j|}|Sr)rrrr@r'rr/rrs rrEzRmsNormAct2d.forwardsd ??4#8#8$++txxPA1d33T[[$((KA IIaL HHQKr rrLs@rrrrr r)NFTr)1rItypingrrrrrRrtorch.nnrr>torchvision.ops.miscr create_actr fast_normr r rrrrnormrr trace_utilsrtorch.nn.functionalr ImportErrorrrr"r[rNr_Modulererrr GroupNormrr LayerNormrrrrrr rrs .- $2(qq$ $, 1\R^^\~ r'' (V=d588??=d@)X&R!2<<!HBLL82<<8R\\:B9w$#$s D D#"D#