kh@ ddlZddlZddlZddlmZddlmZdZddZddZddZ dZ dd ejd e d e d e d e f dZ dd ejd e d e d e fdZy)N)nn)_calculate_fan_in_and_fan_outcd}||d|zz ks ||d|zzkDrtjdd|||z |z }|||z |z }|jd|zdz d|zdz |j|j |t j dz|j||j|||S)Ncddtj|tjdz zdz S)N?@)matherfsqrt)xs S/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/layers/weight_init.pynorm_cdfz _trunc_normal_..norm_cdf s(TXXa$))B-/00B66zjmean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect.) stacklevelr)minmax) warningswarnuniform_erfinv_mul_r r add_clamp_)tensormeanstdabrlus r _trunc_normal_r#s7 q1s7{q1s7{ 2 E!" $ !d(c!"A!d(c!"A OOAEAIq1uqy) NN KKdiim#$ KK MMaQM Mrcttj5t|||||cdddS#1swYyxYw)aFills the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distribution :math:`\mathcal{N}(\text{mean}, \text{std}^2)` with values outside :math:`[a, b]` redrawn until they are within the bounds. The method used for generating the random values works best when :math:`a \leq \text{mean} \leq b`. NOTE: this impl is similar to the PyTorch trunc_normal_, the bounds [a, b] are applied while sampling the normal with mean/std applied, therefore a, b args should be adjusted to match the range of mean, std args. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution a: the minimum cutoff value b: the maximum cutoff value Examples: >>> w = torch.empty(3, 5) >>> nn.init.trunc_normal_(w) N)torchno_gradr#rrrrr s r trunc_normal_r(+s2. 7fdCA6777s.7ctj5t|dd|||j|j |ddd|S#1swY|SxYw)aFills the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distribution :math:`\mathcal{N}(\text{mean}, \text{std}^2)` with values outside :math:`[a, b]` redrawn until they are within the bounds. The method used for generating the random values works best when :math:`a \leq \text{mean} \leq b`. NOTE: this 'tf' variant behaves closer to Tensorflow / JAX impl where the bounds [a, b] are applied when sampling the normal distribution with mean=0, std=1.0 and the result is subsequently scaled and shifted by the mean and std args. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution a: the minimum cutoff value b: the maximum cutoff value Examples: >>> w = torch.empty(3, 5) >>> nn.init.trunc_normal_(w) rrN)r%r&r#rrr's r trunc_normal_tf_r*FsR. $vq#q!, Cd#$ M$ Ms 0AAc,t|\}}|dk(r|}n|dk(r|}n |dk(r||zdz }|z }|dk(r$t|tj|dz y|dk(rCt j 5|j tj|dddy|d k(rItjd |z}t j 5|j| |dddytd |#1swYyxYw#1swYyxYw) Nfan_infan_outfan_avgrtruncated_normalg۶%?rnormaluniformzinvalid distribution ) rr*r r r%r&normal_r ValueError) rscalemode distributionr,r-denomvariancebounds r variance_scaling_r<cs3F;OFG x    '!Q&u}H))TYYx%8;M%MN  ! ]]_ 4 NNtyy2N 3 4 4  " !h,' ]]_ + OOUFE * + +0?@@ 4 4 + +s3&C>D >D Dc t|ddy)Nr,r/)r7r8)r<)rs r lecun_normal_r>|sf8:LMrmodulename init_bias head_biasclassifier_namect|tjtjtjtj fr|j |rTtjj|jtjj|j|ytjj|jdt|tjr8|j+tjj|j|yyyt|dr|jyy)N{Gz?r0 init_weights) isinstancerLinearConv1dConv2dConv3d startswithinitzeros_weight constant_biasr(hasattrrF)r?r@rArBrCs r init_weight_vitrSs&299biiBIIFG ??? + GGNN6== ) GG  fkk9 5 GG ! !&--T ! :&")),1H!!&++y92I,  ( )rc&t|tjr|j|rTtjj |j tjj|j|ytjj|j |jYd|vr+tjj|jdn(tjj |jyyt|tjrLt|j |j*tjj |jyyt|dr|jyy)Nmlpgư>r0rF)rGrrHrLrMrNrOrPrQxavier_uniform_r4rJr>rRrF)r?r@rBrCs r init_weight_jaxrWs &"))$ ??? + GGNN6== ) GG  fkk9 5 GG # #FMM 2{{&:?4- 6RWW^^\b\g\gMh' FBII &fmm$ ;; " GGNN6;; ' #  ( )r)rgr)rr,r1)rErXhead)rXrY)r%r rr torch.nn.initrr#r(r*r<r>ModulestrfloatrSrWrr r_s 7 F76:A2N %     ,%   r