kh= dZddlZddlmZddlmcmZddlmZ dde de de de d e d e f d Z dd ejde de de de d e f d ZGddejZdde de de fdZGddejZy)a> DropBlock, DropPath PyTorch implementations of DropBlock and DropPath (Stochastic Depth) regularization layers. Papers: DropBlock: A regularization method for convolutional networks (https://arxiv.org/abs/1810.12890) Deep Networks with Stochastic Depth (https://arxiv.org/abs/1603.09382) Code: DropBlock impl inspired by two Tensorflow impl that I liked: - https://github.com/tensorflow/tpu/blob/master/models/official/resnet/resnet_model.py#L74 - https://github.com/clovaai/assembled-cnn/blob/master/nets/blocks.py Hacked together by / Copyright 2020 Ross Wightman N)ndgrid drop_prob block_size gamma_scale with_noiseinplace batchwisec.|j\}}} } | | z} t|t| | } ||z| z| dzz | |z dz| |z dzzz } ttj| |j tj| |j \}}|| dzk\|| | dz dzz kz|| dzk\|| | dz dzz kzz}tj |dd| | fj|j}|r1tjd|| | f|j|j }ntj|}d| z |z |zdk\j|j}tj| | d| dz }|r|r0tjd|| | f|j|j ntj|}|r(|j|j!|d|z z|S||z|d|z zz}|S|j#|jtj$j'j)dz j|j}|r|j||z|S||z|z}|S)a  DropBlock. See https://arxiv.org/pdf/1810.12890.pdf DropBlock with an experimental gaussian noise option. This layer has been tested on a few training runs with success, but needs further validation and possibly optimization for lower runtime impact. r)devicedtype)rr  kernel_sizestridepaddinggHz>)shapeminrtorcharanger reshapetorrand rand_likeF max_pool2drandn randn_likemul_add_numelfloat32sumadd)xrrrrr r BCHW total_sizeclipped_block_sizegammaw_ih_i valid_block uniform_noise block_mask normal_noisenormalize_scales L/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/layers/drop.py drop_block_2dr6sJAq!QQJZQ3 ) #j 03E3J J ^a A NQ$6 7 9Eell1QXX6 Qqxx8XYHC-22sQBTWXBX]^A^=^7^_-22sQBTWXBX]^A^=^7^_aK-- aAq\:==AGG=LK Aq!Q)rrr empty_like bernoulli_rrrrnormal_r r!r"r#r$r%)r&rrrrr r'r(r)r*r+r,r-r2r3r4s r5drop_block_fast_2dr=NsJAq!QQJZQ3 ) #j 03E3J J ^a A NQ$6 7 9E!!!$//6J agg,>qRdhiRikJ''*224  FF2 ? # ( ( )B C HR*_% z(AAA H ^ %++- EMM 0R0V0V0X0\0\]a0bbffmnmtmtfu  FF:/ 0 HJ0A Hr7cVeZdZdZ d dededededededeffd Zd ZxZ S) DropBlock2dz9 DropBlock. See https://arxiv.org/pdf/1810.12890.pdf rrrrr r fastctt| ||_||_||_||_||_||_||_ yN) superr?__init__rrrrr r r@) selfrrrrr r r@ __class__s r5rDzDropBlock2d.__init__ysD k4)+"&$$ " r7c l|jr |js|S|jrBt||j|j|j |j |jSt||j|j|j |j |j|jSrB) trainingrr@r=rrrr r6r rEr&s r5forwardzDropBlock2d.forwards}}DNNH 99%4>>4??D4D4DdooW[WcWce e!4>>4??D4D4DdooW[WcWceiesesu ur7)皙?r9FFFT) __name__ __module__ __qualname____doc__floatintboolrDrJ __classcell__rFs@r5r?r?usn #!$$!#     $ur7r?rH scale_by_keepc|dk(s|s|Sd|z }|jdfd|jdz zz}|j|j|}|dkDr|r|j |||zS)a(Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rr)r)rndim new_emptyr;div_)r&rrHrV keep_probr random_tensors r5 drop_pathr^swBhI I WWQZMDAFFQJ/ /EKK&11)ros     3 3 3  3  3  3 3 p   $ <<$ $ $  $  $  $ Nu"))uBE$t( ;ryy ;r7