khHdZddlmZddlmZddlZddlmZddlmZm Z ddl m Z ddl m Z mZmZd d lmZd d lmZmZmZd gZGd d e Zd\dZd]dZeidedddedddedddedddddeddd d!edd"d d#edd$d d%edd&ddd'd(edd)d*edd+ddd,edd-d.edd/d0edd1ddd2edd3d4edd5ddd6edd7d8edd9d:edd;ddeddddedd?d:edd@ddeddAd:dBZed^dCe fdDZed^dCe fdEZed^dCe fdFZed^dCe fdGZed^dCefdHZed^dCefdIZ ed^dCefdJZ!ed^dCefdKZ"ed^dCe fdLZ#ed^dCe fdMZ$ed^dCe fdNZ%ed^dCe fdOZ&ed^dCe fdPZ'ed^dCe fdQZ(ed^dCe fdRZ)ed^dCe fdSZ*ee+d8dTdUdVdWdXdYdZd[y)_a[ DeiT - Data-efficient Image Transformers DeiT model defs and weights from https://github.com/facebookresearch/deit, original copyright below paper: `DeiT: Data-efficient Image Transformers` - https://arxiv.org/abs/2012.12877 paper: `DeiT III: Revenge of the ViT` - https://arxiv.org/abs/2204.07118 Modifications copyright 2021, Ross Wightman )partial)OptionalN)nnIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)resample_abs_pos_embed)VisionTransformer trunc_normal_checkpoint_filter_fn)build_model_with_cfg)generate_default_cfgsregister_modelregister_model_deprecationsVisionTransformerDistilledc8eZdZdZfdZdfd ZejjddZ ejjde jfdZ dde deefd Zejjdd Zd Zdd edej(fd ZxZS)rz Vision Transformer w/ Distillation Token and Head Distillation token & head support for `DeiT: Data-efficient Image Transformers` - https://arxiv.org/abs/2012.12877 cx|jdd}t||i|ddi|jdvsJd|_t j tjdd|j|_ t j tjd|jj|jz|j|_ |jdkDr*t j|j|jnt j |_d|_|j'|y) N weight_initskip)tokenr rF)popsuper__init__ global_poolnum_prefix_tokensr Parametertorchzeros embed_dim dist_token patch_embed num_patches pos_embed num_classesLinearIdentity head_distdistilled_training init_weights)selfargskwargsr __class__s L/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/models/deit.pyrz#VisionTransformerDistilled.__init__#sjj3  $=&=f=:---!",,u{{1a'HI KK4++77$:P:PPRVR`R` acHLHXHX[\H\4>>43C3CDbdbmbmbo"' +&cTt|jdt| |y)Ng{Gz?)std)mode)r r#rr,)r-r5r0s r1r,z'VisionTransformerDistilled.init_weights1s!doo3/ $'r2c tdddgS)Nz+^cls_token|pos_embed|patch_embed|dist_token)z^blocks\.(\d+)N)z^norm)i)stemblocks)dict)r-coarses r1 group_matcherz(VisionTransformerDistilled.group_matcher5s?)$&  r2returnc2|j|jfSNheadr*)r-s r1get_classifierz)VisionTransformerDistilled.get_classifier>syy$..((r2r'rc*||_|dkDr tj|j|ntj|_|dkDr0tj|j|j|_ytj|_y)Nr)r'rr(r"r)r@r*)r-r'rs r1reset_classifierz+VisionTransformerDistilled.reset_classifierBse&>IAoBIIdnnk:SUS^S^S` HSVW4>>43C3CD]_]h]h]jr2c||_yr>)r+)r-enables r1set_distilled_trainingz1VisionTransformerDistilled.set_distilled_trainingGs "(r2c|jrn|j\}}}}|jj}t |j ||f||j rdn |j}|j|d|}n |j }|j rp||z}tj|jj|jddd|jj|jddd|fd}notj|jj|jddd|jj|jddd|fd}||z}|j|S)Nr)new_sizeold_sizerr )dim)dynamic_img_sizeshaper$ grid_sizer r&no_embed_classrviewr cat cls_tokenexpandr#pos_drop)r-xBHWCprev_grid_sizer&s r1 _pos_embedz%VisionTransformerDistilled._pos_embedKsV  JAq!Q!--77N.Q''+':':!@V@V I q"a AI   I A %%aggaj"b9&&qwwqz2r: A %%aggaj"b9&&qwwqz2r: A I A}}Qr2 pre_logitsc|dddf|dddf}}|r||zdz S|j|}|j|}|jr.|jr"tj j s||fS||zdz S)Nrr r)r@r*r+trainingr jit is_scripting)r-rUr\x_dists r1 forward_headz'VisionTransformerDistilled.forward_headlsadGQq!tW6 J!# # IIaL'  " "t}}UYY=S=S=Uf9 J!# #r2rFr>)T)__name__ __module__ __qualname____doc__rr,r r_ignorer;rModulerAintrstrrCrFr[boolTensorrb __classcell__)r0s@r1rrs '( YY   YY) ))kCkhsmk  YY)) B $$ $5<< $r2c |jdd}|rtnt}t|||ft t dt |dd|}|S)N out_indicesT)adapt_layer_scalegetter)rq feature_cls)pretrained_filter_fn feature_cfg)rrr rrr r9)variant pretrained distilledr/rq model_clsmodels r1 _create_deitr}zs]**]A.K.7*=NI  %%9TR[hG     E Lr2c 2|ddddddttddd |S) Ni)rrrg?bicubicTzpatch_embed.projr@) urlr' input_size pool_sizecrop_pct interpolationfixed_input_sizemeanr4 first_conv classifierr)rr/s r1_cfgrs2=t%.B(     r2zdeit_tiny_patch16_224.fb_in1kztimm/zFhttps://dl.fbaipublicfiles.com/deit/deit_tiny_patch16_224-a1311bcf.pth) hf_hub_idrzdeit_small_patch16_224.fb_in1kzGhttps://dl.fbaipublicfiles.com/deit/deit_small_patch16_224-cd65a155.pthzdeit_base_patch16_224.fb_in1kzFhttps://dl.fbaipublicfiles.com/deit/deit_base_patch16_224-b5f2ef4d.pthzdeit_base_patch16_384.fb_in1kzFhttps://dl.fbaipublicfiles.com/deit/deit_base_patch16_384-8de9b5d1.pth)rrrg?)rrrrz'deit_tiny_distilled_patch16_224.fb_in1kzPhttps://dl.fbaipublicfiles.com/deit/deit_tiny_distilled_patch16_224-b40b3cf7.pthr?)rrrz(deit_small_distilled_patch16_224.fb_in1kzQhttps://dl.fbaipublicfiles.com/deit/deit_small_distilled_patch16_224-649709d9.pthz'deit_base_distilled_patch16_224.fb_in1kzPhttps://dl.fbaipublicfiles.com/deit/deit_base_distilled_patch16_224-df68dfff.pthz'deit_base_distilled_patch16_384.fb_in1kzPhttps://dl.fbaipublicfiles.com/deit/deit_base_distilled_patch16_384-d0272ac0.pth)rrrrrzdeit3_small_patch16_224.fb_in1kz;https://dl.fbaipublicfiles.com/deit/deit_3_small_224_1k.pthzdeit3_small_patch16_384.fb_in1kz;https://dl.fbaipublicfiles.com/deit/deit_3_small_384_1k.pthz deit3_medium_patch16_224.fb_in1kzhttps://dl.fbaipublicfiles.com/deit/deit_3_huge_224_21k_v1.pth)(deit3_small_patch16_384.fb_in22k_ft_in1k)deit3_medium_patch16_224.fb_in22k_ft_in1k'deit3_base_patch16_224.fb_in22k_ft_in1k'deit3_base_patch16_384.fb_in22k_ft_in1k(deit3_large_patch16_224.fb_in22k_ft_in1k(deit3_large_patch16_384.fb_in22k_ft_in1k'deit3_huge_patch14_224.fb_in22k_ft_in1kr<c Rtdddd}tdd|it|fi|}|S)z DeiT-tiny model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit.  rr patch_sizer"depth num_headsry)deit_tiny_patch16_224r9r}ryr/ model_argsr|s r1rrs7 s"JJ  fZ f4PZKe^dKe fE Lr2c Rtdddd}tdd|it|fi|}|S)z DeiT-small model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrry)deit_small_patch16_224rrs r1rrs7 s"JJ  gj gDQ[Lf_eLf gE Lr2c Rtdddd}tdd|it|fi|}|S)z DeiT base model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrry)deit_base_patch16_224rrs r1rr7 s"KJ  fZ f4PZKe^dKe fE Lr2c Rtdddd}tdd|it|fi|}|S)z DeiT base model @ 384x384 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrry)deit_base_patch16_384rrs r1rrrr2c Vtdddd}t d|ddt|fi|}|S) z DeiT-tiny distilled model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrrrTryrz)deit_tiny_distilled_patch16_224rrs r1rrsG s"JJ ) p6@D pTXYcTngmTn pE Lr2c Vtdddd}t d|ddt|fi|}|S) z DeiT-small distilled model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrrTr) deit_small_distilled_patch16_224rrs r1rr'sG s"JJ * q7AT qUYZdUohnUo qE Lr2c Vtdddd}t d|ddt|fi|}|S)z DeiT-base distilled model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrTr)deit_base_distilled_patch16_224rrs r1rr2G s"KJ ) p6@D pTXYcTngmTn pE Lr2c Vtdddd}t d|ddt|fi|}|S)z DeiT-base distilled model @ 384x384 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrTr)deit_base_distilled_patch16_384rrs r1rr=rr2c Vtdddddd}td d|it|fi|}|S) z DeiT-3 small model @ 224x224 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrTư>rr"rrrO init_valuesry)deit3_small_patch16_224rrs r1rrH= s"Z^lpqJ  hz hTR\Mg`fMg hE Lr2c Vtdddddd}td d|it|fi|}|S) z DeiT-3 small model @ 384x384 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrTrrry)deit3_small_patch16_384rrs r1rrRrr2c Vtdddddd}td d|it|fi|}|S) z DeiT-3 medium model @ 224x224 (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rirTrrry)deit3_medium_patch16_224rrs r1rr\s= s"Z^lpqJ  i idS]NhagNh iE Lr2c Vtdddddd}tdd|it|fi|}|S) z DeiT-3 base model @ 224x224 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrTrrry)deit3_base_patch16_224rrs r1rrf= s"[_mqrJ  gj gDQ[Lf_eLf gE Lr2c Vtdddddd}tdd|it|fi|}|S) DeiT-3 base model @ 384x384 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrTrrry)deit3_base_patch16_384rrs r1rrprr2c Vtdddddd}tdd|it|fi|}|S) z DeiT-3 large model @ 224x224 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rTrrry)deit3_large_patch16_224rrs r1rrz= t2\`nrsJ  hz hTR\Mg`fMg hE Lr2c Vtdddddd}tdd|it|fi|}|S) z DeiT-3 large model @ 384x384 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrTrrry)deit3_large_patch16_384rrs r1rrrr2c Vtdddddd}td d|it|fi|}|S) ri rTrrry)deit3_huge_patch14_224rrs r1rrs= t2\`nrsJ  gj gDQ[Lf_eLf gE Lr2rrrrrrr) deit3_small_patch16_224_in21ft1k deit3_small_patch16_384_in21ft1k!deit3_medium_patch16_224_in21ft1kdeit3_base_patch16_224_in21ft1kdeit3_base_patch16_384_in21ft1k deit3_large_patch16_224_in21ft1k deit3_large_patch16_384_in21ft1kdeit3_huge_patch14_224_in21ft1k)FFrcrd),rh functoolsrtypingrr r timm.datarr timm.layersr timm.models.vision_transformerr r r _builderr _registryrrr__all__rr}r default_cfgsrrrrrrrrrrrrrrrrrer2r1rs  A.aa*YY ' ([$!2[$| %^&#T T&V^& %d U'W ^&$T T&V^&$T T 3&0^& .t ^(0*!^&(/ _(1*)^&0.t ^(0*1^&8.t ^ 3( 0*9^&D&t I(KE^&J&t I 3(0K^&R' J)LS^&X%d H'JY^&^%d H 3'0_^&f&t I(Kg^&l&t I 3(0m^&t%d H'Ju^&|/ J1}^&D15 J 31026 K204 I004 I 30015 J115 J 31004 L0u^&^ B9J:K9J9JC]D^C]C];L;L