kh-dZddlZddlmZddlZddlmZddlmcmZ ddl m Z m Z m Z mZmZmZmZmZ ddlmZGddej*ZGd d ej*ZGd d ej.ZGd dej.Zdej2defdZdej2dej2dej2defdZdej2dej2dej2defdZGddej.ZGddej@Z!Gddej@Z"Gddej@Z#Gd d!ej@Z$y#e$r ddl mZY2wxYw)"z Normalization layers and wrappers Norm layer definitions that support fast norm and consistent channel arg order (always first arg). Hacked together by / Copyright 2022 Ross Wightman N)Tuple) is_fast_normfast_group_normfast_layer_norm fast_rms_norm rms_norm2dfast_rms_norm2dfast_simple_norm simple_norm)rms_normc`eZdZUejj eed<dfd ZdZ xZ S) GroupNorm _fast_normcJt|||||t|_y)N)epsaffinesuper__init__rr)self num_channels num_groupsrr __class__s L/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/layers/norm.pyrzGroupNorm.__init__s" \s6J&.c |jr7t||j|j|j|j St j||j|j|j|j SNrrrweightbiasrF group_normrxs rforwardzGroupNorm.forward!S ??"1doot{{DIItxxX X<<4??DKKDHHU Ur) gh㈵>T) __name__ __module__ __qualname__torchjitFinalbool__annotations__rr& __classcell__rs@rrrs! %%) VrrceZdZUdZej j eed<fdZ dejdejfdZ xZ S) GroupNorm1zL Group Normalization with 1 group. Input: tensor in shape [B, C, *] rc Ft|d|fi|t|_y)Nrr)rrkwargsrs rrzGroupNorm1.__init__.s  L3F3&.rr%returnc |jr7t||j|j|j|j St j||j|j|j|j Srrr$s rr&zGroupNorm1.forward2r'r r)r*r+__doc__r,r-r.r/r0rTensorr&r1r2s@rr4r4(s@ %%)VV%,,Vrr4ceZdZUdZej j eed<dfd Z dejdejfdZ xZ S) LayerNormz# LayerNorm w/ fast norm option rcHt||||t|_yN)relementwise_affinerrrrrrs rrzLayerNorm.__init__>  36J&.rr%r7c|jr9t||j|j|j|j }|St j||j|j|j|j }|Sr)rrnormalized_shaper r!rr" layer_normr$s rr&zLayerNorm.forwardBsf ??4#8#8$++tyyRVRZRZ[A Q 5 5t{{DIItxxXArư>Tr9r2s@rr=r=9s; %%)%,,rr=ceZdZUdZej j eed<dfd Z dejdejfdZ xZ S) LayerNorm2dz5 LayerNorm for channels of '2D' spatial NCHW tensors rcHt||||t|_yr?rrAs rrzLayerNorm2d.__init__NrBrr%r7c`|jdddd}|jr8t||j|j|j |j }nAtj||j|j|j |j }|jdddd}|SNrr) permuterrrDr r!rr"rEr$s rr&zLayerNorm2d.forwardRs IIaAq ! ??4#8#8$++tyyRVRZRZ[A Q 5 5t{{DIItxxXA IIaAq !rrFr9r2s@rrIrIJs9? %%)%,,rrItensorr7ctjjr|jS|jtjS)N) memory_format)r,r- is_scripting is_contiguouscontiguous_format)rPs r_is_contiguousrV\s; yy##%%##%2I2I#JJrr%r r!rctj|ddd\}}||z tj||zz}||ddddfz|ddddfz}|S)NrFT)dimunbiasedkeepdim)r,var_meanrsqrt)r%r r!rsus r_layer_norm_cfr_ds_ >>!UD ADAq Q%++a#g&&A F1dD= !!DD$$77A Hrc|jdd}||zjdd||zz jd}||z tj||zz}||j ddddz|j ddddz}|S)NrT)rXrZr)meanclampr,r\view)r%r r!rr^r]s r_layer_norm_cf_sqmreks 1d#A a%!T *a!e 4;;A>A Q%++a#g&&A FKK2q! $$tyyB1'==A HrcDeZdZdZdfd Zdej fdZxZS)LayerNormExp2da_ LayerNorm for channels_first tensors with 2d spatial dimensions (ie N, C, H, W). Experimental implementation w/ manual norm for tensors non-contiguous tensors. This improves throughput in some scenarios (tested on Ampere GPU), esp w/ channels_last layout. However, benefits are not always clear and can perform worse on other GPUs. c(t|||y)N)r)rr)rrrrs rrzLayerNormExp2d.__init__|s 3/rr7cBt|rgtj|jdddd|j|j |j |jjdddd}|St||j |j |j}|SrL) rVr"rErOrDr r!rr_r$s rr&zLayerNormExp2d.forwards !   !Q1%t'<'rsZ   $, V VVV" "",,$K5<<KDK ell ELL   SX  %,,   ELL W\ R\\*'bii'T+ +\%%P'299'g$##$sE E.-E.