khdZddlZddlmZddlmcmZddlmZm Z m Z GddejZ GddejZ Gd d ejZGd d ejZy) a Convolution with Weight Standardization (StdConv and ScaledStdConv) StdConv: @article{weightstandardization, author = {Siyuan Qiao and Huiyu Wang and Chenxi Liu and Wei Shen and Alan Yuille}, title = {Weight Standardization}, journal = {arXiv preprint arXiv:1903.10520}, year = {2019}, } Code: https://github.com/joe-siyuan-qiao/WeightStandardization ScaledStdConv: Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 Official Deepmind JAX code: https://github.com/deepmind/deepmind-research/tree/master/nfnets Hacked together by / copyright Ross Wightman, 2021. N) get_paddingget_padding_valuepad_samec.eZdZdZ dfd ZdZxZS) StdConv2dzConv2d with Weight Standardization. Used for BiT ResNet-V2 models. Paper: `Micro-Batch Training with Batch-Channel Normalization and Weight Standardization` - https://arxiv.org/abs/1903.10520v2 c `| t|||}t | ||||||||| |_y)Nstridepaddingdilationgroupsbias)rsuper__init__eps) self in_channel out_channels kernel_sizer r r rrr __class__s P/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/layers/std_conv.pyrzStdConv2d.__init__ sF ?!+vx@G   k&hvD  Jc ftj|jjd|jddddd|j j |j}tj|||j|j|j|j|j}|SNrT)trainingmomentumr) F batch_normweightreshaperr reshape_asconv2drr r r rrxr"s rforwardzStdConv2d.forward*s KK  4#4#4b 94BDHH66@j6M  HHQ 4;; dmmUYU`U` ar)rNrrFư>__name__ __module__ __qualname____doc__rr( __classcell__rs@rrrs LP26rrc.eZdZdZ dfd ZdZxZS) StdConv2dSamezConv2d with Weight Standardization. TF compatible SAME padding. Used for ViT Hybrid model. Paper: `Micro-Batch Training with Batch-Channel Normalization and Weight Standardization` - https://arxiv.org/abs/1903.10520v2 c tt||||\}} t | ||||||||| |_| |_y)Nr r r )rrrsame_padr) rrrrr r r rrr is_dynamicrs rrzStdConv2dSame.__init__8sP0V^fg   k&'\d  &# rc |jr,t||j|j|j}t j |jjd|jddddd|jj|j}t j|||j|j|j|j|j}|Sr)r5rrr r r r!r"r#rrr$r%rr rr&s rr(zStdConv2dSame.forwardBs ==D,,dkk4==IA KK  4#4#4b 94BDHH66@j6M  HHQ 4;; dmmUYU`U` ar)rSAMErrFr)r*r0s@rr2r22s LR26rr2c.eZdZdZ dfd ZdZxZS)ScaledStdConv2da4Conv2d layer with Scaled Weight Standardization. Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 NOTE: the operations used in this impl differ slightly from the DeepMind Haiku impl. The impact is minor. c (| t|||}t | ||||||||tjt j |jdddf| |_| |jdjdzz|_ | |_ y)Nr rr) rrrnn Parametertorchfullrgainr"numelscaler) r in_channelsrrr r r rrgammar gain_initrs rrzScaledStdConv2d.__init__Us ?!+vx@G  {67]e  &LLT->->1a,H)!TU T[[^113t;; rc tj|jjd|jddd|j |j zjddd|jj|j}tj|||j|j|j|j|jSNrrTr)r"rrr)r r!r"r#rrArCviewrr$r%rr r r rr&s rr(zScaledStdConv2d.forwardas KK  4#4#4b 94II *004BDHH67Aj6M xx6499dkk4<<X\XcXcddr)rNrrT?r)rJr*r0s@rr:r:LsMQLO err:c.eZdZdZ dfd ZdZxZS)ScaledStdConv2dSamea\Conv2d layer with Scaled Weight Standardization and Tensorflow-like SAME padding support Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 NOTE: the operations used in this impl differ slightly from the DeepMind Haiku impl. The impact is minor. c <t||||\}} t | ||||||||tjt j |jdddf| |_| |jdjdzz|_ | |_ | |_ y)Nr4r rrr<)rrrr=r>r?r@rrAr"rBrCr5r)rrDrrr r r rrrErrFr6rs rrzScaledStdConv2dSame.__init__rs0V^fg  {67]e  &LLT->->1a,H)!TU T[[^113t;; " rc  |jr,t||j|j|j}t j |jjd|jddd|j|jzjddd|jj|j}t j|||j |j|j"|j|j$SrH)r5rrr r r r!r"r#rrArCrIrr$r%rr rr&s rr(zScaledStdConv2dSame.forward~s ==D,,dkk4==IA KK  4#4#4b 94II *004BDHH67Aj6M xx6499dkk4<<X\XcXcddr)rr8rrTrJr)rJr*r0s@rrLrLisMSLO errL)r.r?torch.nnr=torch.nn.functional functionalr r rrrConv2drr2r:rLrrrTsb$ == 0BII4ebiie:e"))er