khdZddlZddlmZddlmZddlmZmZm Z m Z m Z m Z m Z mZmZddlmZddlZddlZddlmZddlZddlmZmZdd lmZmZmZdd lmZdd lm Z dd l!m"Z"dd l#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4ddl5m6Z6ddl7m8Z8ddl9m:Z:ddl;mZ>ddl?m@Z@ddlAmBZBddlCmDZDddlEmFZFddlGmHZHdd lImJZJejeLZMd!eNd"efd#ZOed$%Gd&d'ZPGd(d)ZQd*eQd"dfd+ZRd*eQd"dfd,ZSd*eQd"dfd-ZTd*eQd"dfd.ZUd*eQd"dfd/ZVd*eQd"dfd0ZWd*eQd"dfd1ZXd*eQd"dfd2ZYeQZZdJd3Z[e[ dKd4eeNe eNfd5e e eNd6e\d"e eeNe eNeNfffd7Z]d8eNd"ePfd9Z^ dLd8eNd:e\d"eeeffd;Z_ dMde ead?ead@eadAe e\dBe\dCe eadDe eejgefdEed"ejjfdFZddGZe dLdHeejefdBe\d"ejjfdIZfy)Nzs Optimizer Factory w/ custom Weight Decay & Layer Decay support Hacked together by / Copyright 2021 Ross Wightman N) dataclass)partial) AnyCallableDictListOptionalSetTupleTypeUnionfnmatch)param_groups_layer_decayparam_groups_weight_decay)ParamsT OptimTypeOptimizerCallable) AdaBelief) Adafactor)AdafactorBigVision) Adahessian)AdamP) AdamWLegacy)Adan)Adopt)Kron)Lamb)LaProp)Lars)Lion) Lookahead)MADGRAD)Mars) NAdamLegacy)NAdamW) NvNovoGrad) RAdamLegacy) RMSpropTF)SGDP)SGDW class_stringreturnc |jdd\}}tj|}t||S#tt f$r}t d|d|d}~wwxYw)z)Dynamically import a class from a string..rzCould not import z: N)rsplit importlib import_modulegetattr ImportErrorAttributeError)r- module_name class_namemodulees U/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/optim/_optim_factory.py _import_classr<-slC"."5"5c1"= Z((5vz**  (C-l^2aSABBCs58AAAT)frozenceZdZUdZeed<eeefed<dZeed<dZ e ed<dZ e ed <dZ e ed <d Z eed <dZe ed <dZeeeefed<y) OptimInfoaLImmutable configuration for an optimizer. Attributes: name: Unique identifier for the optimizer opt_class: The optimizer class description: Brief description of the optimizer's characteristics and behavior has_eps: Whether the optimizer accepts epsilon parameter has_momentum: Whether the optimizer accepts momentum parameter has_betas: Whether the optimizer accepts a tuple of beta parameters num_betas: number of betas in tuple (valid IFF has_betas = True) defaults: Optional default parameters for the optimizer name opt_class descriptionThas_epsF has_momentum has_betas num_betas second_orderNdefaults)__name__ __module__ __qualname____doc__str__annotations__r rrCrDboolrErFrHintrIrJr rrr;r?r?8sp  IS)^$$KGTL$ItIsL$)-HhtCH~&-rTr?ceZdZdZddZdeddfdZdededdfd Zd eddfd Z d d e ee efd e e ede de e eeeefffdZd edefdZ d!de eefde de eeffdZ d"de ej*efdede ededede e de de ede eej*gefdedej6j8fdZy)#OptimizerRegistryzRegistry managing optimizer configurations and instantiation. This class provides a central registry for optimizer configurations and handles their instantiation with appropriate parameter groups and settings. r.Nc"i|_dh|_y)Nlion) _optimizers_foreach_defaults)selfs r;__init__zOptimizerRegistry.__init__Xs13,28rTinfoc|jj}||jvrtj d|d||j|<y)zRegister an optimizer configuration. Args: info: The OptimInfo configuration containing name, type and description Optimizer z already registered, overwritingN)r@lowerrY_loggerwarning)r[r]r@s r;registerzOptimizerRegistry.register\sH yy  4## # OOj.NO P!%rTaliastargetc|j}||jvrtd||j||j|j<y)zRegister an alias for an existing optimizer. Args: alias: The alias name target: The target optimizer name Raises: KeyError: If target optimizer doesn't exist z/Cannot create alias for non-existent optimizer N)r`rYKeyError)r[rdres r;register_aliasz OptimizerRegistry.register_aliasgsQ )) )LVHUV V*.*:*:6*B'rTr@cV|jj|jy)z4Register an optimizer as defaulting to foreach=True.N)rZaddr`r[r@s r;register_foreach_defaultz*OptimizerRegistry.register_foreach_defaultvs ""4::<0rTfilterexclude_filterswith_descriptionc t|jj}|rLt|tr|g}n|}t }|D] |j fd|Dt|}|r#|D]}|Dcgc]}t||r|}} |r)|D cgc]} | |j| jfc} S|Scc}wcc} w)aList available optimizer names, optionally filtered. Args: filter: Wildcard style filter string (e.g., 'adam*') exclude_filters: Optional list of wildcard patterns to exclude with_description: If True, return tuples of (name, description) Returns: List of either optimizer names or (name, description) tuples c3<K|]}t|s|ywNr).0nfs r; z4OptimizerRegistry.list_optimizers..s%HA'!Q-a%Hs) sortedrYkeys isinstancerOsetupdaterrC) r[rmrnronamesfiltersfiltered_namesexclude_filterrtr@rus @r;list_optimizersz!OptimizerRegistry.list_optimizerszs t'',,./ &#&!(  UN I%%%H%HH I>*E "1 M$)LqN1KLL M KPQ4T4++D1==>Q Q MRs>CC"C cz|j}||jvrtd|d|j|S)zGet the OptimInfo for an optimizer. Args: name: Name of the optimizer Returns: OptimInfo configuration Raises: ValueError: If optimizer is not found r_z not found in registry)r`rY ValueErrorrks r;get_optimizer_infoz$OptimizerRegistry.get_optimizer_infosCzz| t'' 'z$/EFG G%%rT name_or_info bind_defaultsct|tr|j|}nt|tsJ|}t|jtr|jj dr->?I##]%&TU[\\] #m%&dekllms0E)E$ E! EE!$ E>- E99E>model_or_paramsoptlr weight_decaymomentumforeachweight_decay_exclude_1d layer_decayparam_group_fnkwargsc  Rt|tjrYt|dd} | r | |} n=|t |||| |} d}n(|r|rt ||| } d}n|j } n|} |jjd} | d}t| d kDr| d d k(nd }|j|}d |i| }|||d<|jr4|jjD]\}}|j|||jr|jd||js|j!dd|j"s|j!dd||jd||j%|d }|| fi|}|r t'|}|S)aCreate an optimizer instance. Args: model_or_params: Model or parameters to optimize opt: Name of optimizer to create lr: Learning rate weight_decay: Weight decay factor momentum: Momentum factor for applicable optimizers foreach: Enable/disable foreach operation weight_decay_exclude_1d: Whether to skip weight decay for 1d params (biases and norm affine) layer_decay: Layer-wise learning rate decay param_group_fn: Optional custom parameter grouping function **kwargs: Additional optimizer-specific arguments Returns: Configured optimizer instance Raises: ValueError: If optimizer not found or configuration invalid no_weight_decayctSrr)rzrSrTr;z4OptimizerRegistry.create_optimizer..s RURWrTN)rrno_weight_decay_listr)rr_rr lookaheadFrrrepsbetasrr)rynnModuler4rr parametersr`splitlenrrJitems setdefaultrErDpoprFrr#)r[rrrrrrrrrrrparams opt_splitopt_name use_lookaheadropt_argskvrA optimizers r;create_optimizerz"OptimizerRegistry.create_optimizersF oryy 1Xgo7H-XZO'8(1#!- +)8,C  " "92#!-)8 " (335%FIIK%%c* R=7:9~7I ! 3u **84%3L#KF#K >HTN    ))//1 *1##Aq) *     H 5 LL %!! LL$ '      7 3,,XU,K f11  !),IrTr.NrBNFT)Nr?NTNN)rKrLrMrNr\r?rcrOrhrlr rr rQr rrrrrrrrfloatrrroptim OptimizerrrSrTr;rVrVQs 4 &Y &4 & CC C C C1S1T1 -/37%* ##tCy.)#&d3i0## # eCsCx() * #J&s&y&(#'1Y/1 1 y++ , 1n#'"$!&*,0+/GKk"299g#56kk k  k  kd^k&*k"%k%Xryyk7.B%CDkk   krTrVregistryc 6tdtjjddddditdtjjddddditd td dddi td t d ddddig}|D]}|j |y)zRegister SGD-based optimizerssgdzDtorch.Optim Stochastic Gradient Descent (SGD) with Nesterov momentumFTnesterovr@rArCrDrErJrzEtorch.Optim Stochastic Gradient Descent (SGD) with classical momentumsgdpz0SGD with built-in projection to unit norm spherer@rArCrErJsgdwz5SGD with decoupled weight decay and Nesterov momentumN)r?rrSGDr+r,rc)rsgd_optimizersrs r;_register_sgd_variantsrQs kkoo^ $'   kkoo_ %(   J $'   O $'  1 NB#rTcPtdtjjddtdtjjddtdt ddtd t d dd dd  tdtjjddtdtddtdtddtdtjjddtdtddtdtjjddddi tdtjjddtdtdtd td!td"td#td$td%d&di'g}|D]}|j!|y())zRegister Adam-based optimizersadamz,torch.optim.Adam, Adaptive Moment EstimationTr@rArCrFadamwz3torch.optim.AdamW, Adam with decoupled weight decay adamwlegacyz !Q.5   R.5   Y"&$G   [#'4H  Z)-N   _)-N   P.5   R.5   f"tW[\   (m.5  CGP$# !Z!%F   d#'DTXY  [!%F   b"&4SWX   Y!%F    (S!%F  K+X"#rTctdtddditdtddditdtd dddi td td dditd t ddddddtdt ddddddtdtddddtdt ddddi tdt ddddd tdtddddi tdtd ddddi!td"td#dddi td$td%dddi td&td'dd(dd)*td+td,ddddd-.g}|D]}|j|y)/N cadafactorzCautious AdafactorrTr cadafactorbvzCautious Big Vision AdafactorcadamwzCautious AdamWrcadoptzCautious Adoptcadanz-Cautious Adaptive Nesterov Momentum AlgorithmF)rno_proxr@rArCrJrFrHcadanwz?Cautious Adaptive Nesterov Momentum with decoupled weight decaycadoptwz!Cautious AdoptW (decoupled decay))rrclambz Cautious LAMBclambwz)Cautious LAMB with decoupled weight decay)rrclapropzCautious LaPropclionz Cautious Lionrcmarsz Cautious MARScnadamwzCautious NAdamW crmsproptfz!Cautious TensorFlow-style RMSpropr)rrrcsgdwz>Cautious SGD with decoupled weight decay and Nesterov momentum)rrr)r?rrrrrrr r"r%r'r*r,rc)rcautious_optimizersrs r;_register_cautious_optimizersrs,&  (7&  !(&   (&  G!%%8   Y!%$7   ;#'D9  '&   C!%$?   )&   '!4(   '&   )&   ;"t4   X"&48  EjV##rTcptdtdddditdtdddditd tjjd td tjj d dditdt dddiddtdt dddiddtdtdddtdtddtdtddd di!td"td#d$td%td&dd'td(td)dtd*td+dd di!td,td-d$td.td/d$td0tjjd1dd2d3i!td4td5dd2d3i!g}|D]}|j!||j#d%y6)7z!Register miscellaneous optimizers adabeliefz7Adapts learning rate based on gradient prediction errorTrectifyFr radabeliefz,Rectified AdaBelief with variance adaptationadadeltazQtorch.optim.Adadelta, Adapts learning rates based on running windows of gradientsradagradzMtorch.optim.Adagrad, Adapts learning rates using cumulative squared gradientsrg:0yE>radanz$Adaptive Nesterov Momentum Algorithmrrradanwz6Adaptive Nesterov Momentum with decoupled weight decay adahessianz"An Adaptive Second Order Optimizer)r@rArCrFrIkronz5PSGD optimizer with Kronecker-factored preconditionerrkronwzPPSGD optimizer with Kronecker-factored preconditioner and decoupled weight decayrrlapropz*Separating Momentum and Adaptivity in AdamrrXz8Evolved Sign Momentum optimizer for improved convergence)r@rArCrDrFmadgradz'Momentum-based Adaptive gradient methodmadgradwz#MADGRAD with decoupled weight decaymarszDUnleashing the Power of Variance Reduction for Training Large Modelsnovogradz3Normalized Adam with L2 norm gradient normalizationrmspropz1torch.optim.RMSprop, Root Mean Square Propagationrr rmsproptfzETensorFlow-style RMSprop implementation, Root Mean Square PropagationN)r?rrrAdadeltaAdagradrrrr r"r$r%r(RMSpropr*rcrl)rother_optimizersrs r;_register_other_optimizersr5s Q'   F&   kk**k kk))gT]  >'   P&    <   O  j'.   D  R   A  ='.   ^   M  kk))Ks^   _s^  Wrf # %%f-rTc tddddddditdd d dd di td d ddd di tddddtddddddi g}|D]}|j|y)z&Register APEX optimizers (lazy import)fusedsgdzapex.optimizers.FusedSGDz8NVIDIA APEX fused SGD implementation for faster trainingFTrr fusedadamzapex.optimizers.FusedAdamz%NVIDIA APEX fused Adam implementation adam_w_moder fusedadamwz&NVIDIA APEX fused AdamW implementation fusedlambzapex.optimizers.FusedLAMBz%NVIDIA APEX fused LAMB implementationr fusednovogradzapex.optimizers.FusedNovoGradz)NVIDIA APEX fused NovoGrad implementationr)gffffff?g\(\?Nr?rc)rapex_optimizersrs r;_register_apex_optimizersr?~s 0R $'   1?#U+   1@#T*   1?   5C|,  ;$OJ#rTchtddddddditdd d dddditd d d dtdd ddtddddtddddtdddddtdddddtddddd !td"d#d$dd !g }|D]}|j|y%)&z.Register bitsandbytes optimizers (lazy import)bnbsgdzbitsandbytes.optim.SGDzbitsandbytes SGDFTrr bnbsgd8bitzbitsandbytes.optim.SGD8bitz0bitsandbytes 8-bit SGD with dynamic quantizationbnbadamzbitsandbytes.optim.Adamzbitsandbytes Adamr bnbadam8bitz1bitsandbytes 8-bit Adam with dynamic quantizationbnbadamwzbitsandbytes.optim.AdamWzbitsandbytes AdamW bnbadamw8bitz2bitsandbytes 8-bit AdamW with dynamic quantizationbnblionzbitsandbytes.optim.Lionzbitsandbytes Lion)rCrDrF bnblion8bitzbitsandbytes.optim.Lion8bitz1bitsandbytes 8-bit Lion with dynamic quantization bnbademamixzbitsandbytes.optim.AdEMAMixzbitsandbytes AdEMAMixr)rCrFrHbnbademamix8bitzbitsandbytes.optim.AdEMAMix8bitz5bitsandbytes 8-bit AdEMAMix with dynamic quantizationNr=)rbnb_optimizersrs r;_register_bnb_optimizersrLs) .* $'   2J $'   /+  /K  0,  0L    %+     )K     )/     -O  }ENL#rTcLttttttt tt tt ttttttjddtjddy)z7Register all default optimizers to the global registry.rr nesterovwrN) rdefault_registryrrr5r?rLrr rhrSrTr;_register_default_optimizersrPsn+,,-()/0./-.!"23()9:##J6##K8rTrmrnroc0tj|||S)a^List available optimizer names, optionally filtered. List all registered optimizers, with optional filtering using wildcard patterns. Optimizers can be filtered using include and exclude patterns, and can optionally return descriptions with each optimizer name. Args: filter: Wildcard style filter string or list of filter strings (e.g., 'adam*' for all Adam variants, or ['adam*', '*8bit'] for Adam variants and 8-bit optimizers). Empty string means no filtering. exclude_filters: Optional list of wildcard patterns to exclude. For example, ['*8bit', 'fused*'] would exclude 8-bit and fused implementations. with_description: If True, returns tuples of (name, description) instead of just names. Descriptions provide brief explanations of optimizer characteristics. Returns: If with_description is False: List of optimizer names as strings (e.g., ['adam', 'adamw', ...]) If with_description is True: List of tuples of (name, description) (e.g., [('adam', 'Adaptive Moment...'), ...]) Examples: >>> list_optimizers() ['adam', 'adamw', 'sgd', ...] >>> list_optimizers(['la*', 'nla*']) # List lamb & lars ['lamb', 'lambc', 'larc', 'lars', 'nlarc', 'nlars'] >>> list_optimizers('*adam*', exclude_filters=['bnb*', 'fused*']) # Exclude bnb & apex adam optimizers ['adam', 'adamax', 'adamp', 'adamw', 'nadam', 'nadamw', 'radam'] >>> list_optimizers(with_description=True) # Get descriptions [('adabelief', 'Adapts learning rate based on gradient prediction error'), ('adadelta', 'torch.optim Adadelta, Adapts learning rates based on running windows of gradients'), ('adafactor', 'Memory-efficient implementation of Adam with factored gradients'), ...] )rOr)rmrnros r;rr sT  + +FOEU VVrTr@c,tj|S)zGet the OptimInfo for an optimizer. Args: name: Name of the optimizer Returns: OptimInfo configuration Raises: ValueError: If optimizer is not found )rOr)r@s r;rr:s  . .t 44rTrc0tj||S)aGet optimizer class by name with option to bind default arguments. Retrieves the optimizer class or a partial function with default arguments bound. This allows direct instantiation of optimizers with their default configurations without going through the full factory. Args: name: Name of the optimizer to retrieve (e.g., 'adam', 'sgd') bind_defaults: If True, returns a partial function with default arguments from OptimInfo bound. If False, returns the raw optimizer class. Returns: If bind_defaults is False: The optimizer class (e.g., torch.optim.Adam) If bind_defaults is True: A partial function with default arguments bound Raises: ValueError: If optimizer name is not found in registry Examples: >>> # Get SGD with nesterov momentum default >>> SGD = get_optimizer_class('sgd') # nesterov=True bound >>> opt = SGD(model.parameters(), lr=0.1, momentum=0.9) >>> # Get raw optimizer class >>> SGD = get_optimizer_class('sgd') >>> opt = SGD(model.parameters(), lr=1e-3, momentum=0.9) r)rOr)r@rs r;rrIsD  / /M / RRrTrrrrrrfilter_bias_and_bnrrrc @tj|f||||||||d| S)a Create an optimizer instance via timm registry. Creates and configures an optimizer with appropriate parameter groups and settings. Supports automatic parameter group creation for weight decay and layer-wise learning rates, as well as custom parameter grouping. Args: model_or_params: A PyTorch model or an iterable of parameters/parameter groups. If a model is provided, parameters will be automatically extracted and grouped based on the other arguments. opt: Name of the optimizer to create (e.g., 'adam', 'adamw', 'sgd'). Use list_optimizers() to see available options. lr: Learning rate. If None, will use the optimizer's default. weight_decay: Weight decay factor. Will be used to create param groups if model_or_params is a model. momentum: Momentum factor for optimizers that support it. Only used if the chosen optimizer accepts a momentum parameter. foreach: Enable/disable foreach (multi-tensor) implementation if available. If None, will use optimizer-specific defaults. filter_bias_and_bn: If True, bias, norm layer parameters (all 1d params) will not have weight decay applied. Only used when model_or_params is a model and weight_decay > 0. layer_decay: Optional layer-wise learning rate decay factor. If provided, learning rates will be scaled by layer_decay^(max_depth - layer_depth). Only used when model_or_params is a model. param_group_fn: Optional function to create custom parameter groups. If provided, other parameter grouping options will be ignored. **kwargs: Additional optimizer-specific arguments (e.g., betas for Adam). Returns: Configured optimizer instance. Examples: >>> # Basic usage with a model >>> optimizer = create_optimizer_v2(model, 'adamw', lr=1e-3) >>> # SGD with momentum and weight decay >>> optimizer = create_optimizer_v2( ... model, 'sgd', lr=0.1, momentum=0.9, weight_decay=1e-4 ... ) >>> # Adam with layer-wise learning rate decay >>> optimizer = create_optimizer_v2( ... model, 'adam', lr=1e-3, layer_decay=0.7 ... ) >>> # Custom parameter groups >>> def group_fn(model): ... return [ ... {'params': model.backbone.parameters(), 'lr': 1e-4}, ... {'params': model.head.parameters(), 'lr': 1e-3} ... ] >>> optimizer = create_optimizer_v2( ... model, 'sgd', param_group_fn=group_fn ... ) Note: Parameter group handling precedence: 1. If param_group_fn is provided, it will be used exclusively 2. If layer_decay is provided, layer-wise groups will be created 3. If weight_decay > 0 and filter_bias_and_bn is True, weight decay groups will be created 4. Otherwise, all parameters will be in a single group )rrrrrrrr)rOr) rrrrrrrTrrrs r;create_optimizer_v2rVnsBV  , ,   ! 2%    rTct|j|j|j|j}t |dd|j |d<t |dd|j|d<t |dd|j|d<t |dd|j|jt |d d|j|d <|S) z cfg/argparse to kwargs helper Convert optimizer args in argparse args or cfg like object to keyword args for updated create fn. )rrrropt_epsNr opt_betasrrr opt_foreachr) dictrrrrr4rXrYrr{rrZ)cfgrs r;optimizer_kwargsr]s GG 66%% F sIt$0 u sK&2--wsM4(4 #}sJ%1 cll#sM4(4OOy MrTmodelc6t|fit|d|iS)zk Legacy optimizer factory for backwards compatibility. NOTE: Use create_optimizer_v2 for new code. )r\rT)rVr])argsr^rTs r;rrs+    t $ . rTrrr)rNrrNTNN)grNlogging dataclassesr functoolsrtypingrrrrr r r r r rr2rtorch.nnr torch.optim _param_groupsrr_typesrrrr rrr adafactor_bvrr'rrrrrr%rrrr(rrrr*r rr!rXr"rr#r+r$r-r%rr&rr' nvnovogradr(rr) rmsprop_tfr*rr+rr, getLoggerrKrarOr<r?rVrrrr rr5r?rLrOrPrQrrrrrrrrVr]rrSrTr;rms]!OOO N99 ," "! '  H %CCC $...0}}@$%6$4$N^&7^D^B;"3;;|{3D{{|m,=m$m^w.):w.tw.t((9(d(VI'8ITIX%&9$ )+/3!&*Wc49n%*W!$s),*W*W %U38_$ %& *WZ 5S 5Y 5"#"S"S"S 9' '("SN" "&#''+CGVryy'12V V UOV V  V $ V!Ve_V!299+w*>!?@VV [[Vr4$( RYY'( !  [[ rT