kh<NdZddlZGddejjZy)z AdaHessian Optimizer Lifted from https://github.com/davda54/ada-hessian/blob/master/ada_hessian.py Originally licensed MIT, Copyright 2020, David Samuel NceZdZdZ dfd ZedZdZdZe jdZ e jd dZ xZ S) Adahessiana Implements the AdaHessian algorithm from "ADAHESSIAN: An Adaptive Second OrderOptimizer for Machine Learning" Arguments: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 0.1) betas ((float, float), optional): coefficients used for computing running averages of gradient and the squared hessian trace (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0.0) hessian_power (float, optional): exponent of the hessian trace (default: 1.0) update_each (int, optional): compute the hessian trace approximation only after *this* number of steps (to save time) (default: 1) n_samples (int, optional): how many times to sample `z` for the approximation of the hessian trace (default: 1) c Ld|kstd|d|kstd|d|dcxkrdksntd|dd|dcxkrdksntd|dd|cxkrdksntd |||_||_| |_d |_t j j|j|_t||||| } tt|3|| |jD]} d| _d|j| d <y) NzInvalid learning rate: zInvalid epsilon value: r?z#Invalid beta parameter at index 0: z#Invalid beta parameter at index 1: zInvalid Hessian power value: i)lrbetaseps weight_decay hessian_power hessian step) ValueError n_samples update_eachavg_conv_kernelseedtorch Generator manual_seed generatordictsuperr__init__ get_paramshessstate) selfparamsr r r r r rrrdefaultsp __class__s Q/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/optim/adahessian.pyrzAdahessian.__init__sAby6rd;< <cz6se<= =eAh$$B58*MN NeAh$$B58*MN Nm*s*<]OLM M"&. *66tyyA%'   j$(:" .AAF,-DJJqM. ) .cy)NTrs r#is_second_orderzAdahessian.is_second_orderFsr$c(d|jDS)zH Gets all parameters in all param_groups with gradients c3NK|]}|dD]}|js|yw)rN) requires_grad).0groupr!s r# z(Adahessian.get_params..Os&]e%/]QQ__]]s% %) param_groupsr's r#rzAdahessian.get_paramsJs ^t00]]r$c|jD]Z}t|jtr|j|d|j zdk(sA|jj \y)z; Zeros out the accumulated hessian traces. rrN)r isinstancerfloatrrzero_)rr!s r# zero_hessianzAdahessian.zero_hessianQsV " Aaffe,A~1NQUQaQa1aef1f  r$c g}td|jD]O}|j|d|jzdk(r|j ||j|dxxdz cc<Qt |dk(ry|j j|djk7r@tj|djj|j|_|Dcgc]}|j}}t|jD]}|Dcgc]D}tjdd|j!|j |jdzd z F}}tj"j|||d ||jdz k }t%|||D]+\}}}|xj&||z|jz z c_-ycc}wcc}w) z} Computes the Hutchinson approximation of the hessian trace and accumulates it for each trainable parameter. c|jduSN)grad)r!s r#z(Adahessian.set_hessian..as!&&"4r$rrrN)rdeviceg@rT) grad_outputs only_inputs retain_graph)filterrrrappendlenrr;rrrrr8rangerrandintsizeautogradzipr) rrr!gradsizsh_zsh_zzs r# set_hessianzAdahessian.set_hessianZs 4doo6GH /Azz!}^,t/?/??1D a JJqM. )Q . ) / v;!   >> F1I$4$4 4"__VAY-=-=>JJ499UDN!'(A((t~~& 3Apvwkl%--1affh$..QRQYQYZ]``cffwBw>>&&vBDqSWSaSadeSeOe'gD r62 3 Q#'DNN22 3  3)xs )GA Gcvd}||}|j|j|jD]}|dD]}|j |j|j rq|j dk(r^tj|jjddgdj|jj|_|jd|d |d zz |j|}t|dk(r5d |d <tj||d <tj||d<|d |d}}|d\}} |d xxdz cc<|j|j!|jd|z |j| j#|j|jd| z d||d zz } d| |d zz } |d} || z j%| dz j!|d} |d | z }|j'|| | |S)z Performs a single optimization step. Arguments: closure (callable, optional) -- a closure that reevaluates the model and returns the loss (default: None) Nrr:T)dimkeepdimrr r rstepexp_avgexp_hessian_diag_sqr )alpha)valuer r )r4rMr/r8rrrQrabsmean expand_asclonemul_rrA zeros_likeadd_addcmul_pow_addcdiv_)rclosurelossr-r!rrTrUbeta1beta2bias_correction1bias_correction2kdenom step_sizes r#rSzAdahessian.stepvs@  9D  &&% =E8_$ =66>QVV^''AEEGqL"YYqvv.33A3MWWXYX^X^_eegAFq5;~)>>>? 1 u:?$%E&M','7'7':E)$383C3CA3FE/0/4Y/?G\A],$W~ uf "  U#((q5y(A#((/88qSXy8Y#$uf '=#= #$uf '=#= /*,/??EEa!eLQQRWX]R^_"$K*::  7E) rrrrFr7)__name__ __module__ __qualname____doc__rpropertyr(rr4rno_gradrMrS __classcell__)r"s@r#rr s&!*.X^U]]_336U]]_55r$r)rnroptim Optimizerrr&r$r#rts'  c&&cr$