kh.2dZddlZddlmZGddeZy)an PyTorch LARS / LARC Optimizer An implementation of LARS (SGD) + LARC in PyTorch Based on: * PyTorch SGD: https://github.com/pytorch/pytorch/blob/1.7/torch/optim/sgd.py#L100 * NVIDIA APEX LARC: https://github.com/NVIDIA/apex/blob/master/apex/parallel/LARC.py Additional cleanup and modifications to properly support PyTorch XLA. Copyright 2021 Ross Wightman N) OptimizercpeZdZdZ dfd ZfdZejddZxZ S)Larsa LARS for PyTorch Paper: `Large batch training of Convolutional Networks` - https://arxiv.org/pdf/1708.03888.pdf Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups. lr (float, optional): learning rate (default: 1.0). momentum (float, optional): momentum factor (default: 0) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) dampening (float, optional): dampening for momentum (default: 0) nesterov (bool, optional): enables Nesterov momentum (default: False) trust_coeff (float): trust coefficient for computing adaptive lr / trust_ratio (default: 0.001) eps (float): eps for division denominator (default: 1e-8) trust_clip (bool): enable LARC trust ratio clipping (default: False) always_adapt (bool): always apply LARS LR adapt, otherwise only when group weight_decay != 0 (default: False) c |dkrtd||dkrtd||dkrtd||r|dks|dk7r tdt|||||||| |  } t | || y)NgzInvalid learning rate: zInvalid momentum value: zInvalid weight_decay value: rz8Nesterov momentum requires a momentum and zero dampening) lrmomentum dampening weight_decaynesterov trust_coeffeps trust_clip always_adapt) ValueErrordictsuper__init__) selfparamsrrr r r r r rrdefaults __class__s K/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/optim/lars.pyrz Lars.__init__#s 86rd;< < c>7zBC C # ;L>JK K Q)q.WX X%#!%   *cjt|||jD]}|jddy)Nr F)r __setstate__ param_groups setdefault)rstategrouprs rrzLars.__setstate__Fs5 U#&& 0E   Z / 0rc d}|$tj5|}ddd|jD]}|d}|d}|d}|d}|d}|d} |dD]k} | j| j} |d k7s|d r| j d } | j d } || z| | |zz| zz }tj | d kDtj | d kD|d d }|d rtj ||dz d }| j| || j||d k7r~|j| }d|vr)tj| jx}|d<n*|d}|j|j| d |z |r| j||} n|} | j| |d n|S#1swYxYw)zPerforms a single optimization step. Args: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nr rr r r r rrrg@?rr)max)alphamomentum_buffer) torch enable_gradrgradnormwhereclampadd_mul_rclonedetachadd)rclosurelossrr rr r r r pr'w_normg_norm trust_ratio param_statebufs rstepz Lars.stepKs  ""$ !y !&&, 1E 0LZ(Hk*IZ(H .K,C8_$ 166>vv 1$n(=VVC[F!YYs^F"-"6&6LCX:X[^:^"_K#(++  FQJ SA#K \*&+kk+d 2KQT&U IIa|I4IIk*q="&**Q-K( ;?D{{4?P?W?W?YYk*;<)*;<*//BN/K#xx8x<"tE$K<0I$ 1, 1\ c ! !s GG) r!rrrFgMbP?g:0yE>FF)N) __name__ __module__ __qualname____doc__rrr%no_gradr8 __classcell__)rs@rrrsL(  !+F0 U]]_99rr)r<r%torch.optim.optimizerrrrrrAs   +t9tr