kh0dZddlZddlmZmZmZmZmZmZm Z m Z ddl Z ddl m Z ddlm cmZddlmZmZddlmZmZmZmZmZmZmZddlmZmZmZmZddl m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'm(Z(d gZ)d e e*e*fde jVfdZ,Gdde jZZ.Gdde jZZ/Gdde jZZ0Gdd e jZZ1d5de2dee2effdZ3e'e3de3ddde3dde3de3ddde3dd de3dde3dee!e3dee!e3ddee"e3dd#ee$e3dd#ee$e3ddee"d% Z4d6d&ee2e jVfd'e jZd(e2d)e5dee2e jVff d*Z6d7d+e2d,e5de1fd-Z7e(d7d,e5de1fd.Z8e(d7d,e5de1fd/Z9e(d7d,e5de1fd0Z:e(d7d,e5de1fd1Z;e(d7d,e5de1fd2Zy)8a BEiT: BERT Pre-Training of Image Transformers (https://arxiv.org/abs/2106.08254) Model from official source: https://github.com/microsoft/unilm/tree/master/beit @inproceedings{beit, title={{BEiT}: {BERT} Pre-Training of Image Transformers}, author={Hangbo Bao and Li Dong and Songhao Piao and Furu Wei}, booktitle={International Conference on Learning Representations}, year={2022}, url={https://openreview.net/forum?id=p-BhZSz59o4} } BEiT-v2 from https://github.com/microsoft/unilm/tree/master/beit2 @article{beitv2, title={{BEiT v2}: Masked Image Modeling with Vector-Quantized Visual Tokenizers}, author={Zhiliang Peng and Li Dong and Hangbo Bao and Qixiang Ye and Furu Wei}, year={2022}, eprint={2208.06366}, archivePrefix={arXiv}, primaryClass={cs.CV} } At this point only the 1k fine-tuned classification weights and model configs have been added, see original source above for pre-training models and procedure. Modifications by / Copyright 2021 Ross Wightman, original copyrights below N)AnyCallableDictListOptionalSetTupleUnion)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) PatchEmbedMlpSwiGLU LayerNormDropPath trunc_normal_use_fused_attn)resample_patch_embedresample_abs_pos_embedresize_rel_pos_bias_tablendgrid)build_model_with_cfg)feature_take_indices) checkpoint)generate_default_cfgsregister_modelBeit window_sizereturnc d|dzdz d|dzdz zdz}|d|dz}tjttj|dtj|d}tj|d}|dddddf|dddddfz }|j dddj }|dddddfxx|ddz z cc<|dddddfxx|ddz z cc<|dddddfxxd|dzdz zcc<tj|dzfdz|j}|jd|ddddf<|dz |dddf<|dz |dddf<|dz |d<|S) aGenerate relative position index for window-based attention. Creates a lookup table for relative position indices between all pairs of positions within a window, including special handling for cls token interactions. Args: window_size: Height and width of the attention window. Returns: Relative position index tensor of shape (window_area+1, window_area+1) where +1 accounts for the cls token. rrN)sizedtype)rr) torchstackrarangeflattenpermute contiguouszerosr%sum)rnum_relative_distance window_areacoordscoords_flattenrelative_coordsrelative_position_indexs L/var/www/teggl/fontify/venv/lib/python3.12/site-packages/timm/models/beit.pygen_relative_position_indexr6<sQ/!3KN8JQ8NORSSa.;q>1K [[ [^ >Z[ \F]]61-N$Q4Z0>!T1*3MMO%--aA6AACOAq!G A 22Aq!G A 22Aq!GKN 2Q 66#kk a/AA/E_MbMbc&5&9&9"&=ABF#%:Q%>AqrE"%:Q%>ABE"$9A$=D! ""ceZdZUdZej j eed< dde de dedede de d e e e e fd e e ffd Z d ejfd Zddejde ejd ejfdZxZS) AttentionzMulti-head attention module with optional relative position bias. Implements multi-head self-attention with support for relative position bias and fused attention operations. Can use either standard or custom head dimensions. fused_attndim num_headsqkv_biasqkv_bias_separate attn_drop proj_dropr attn_head_dimc t |||_||z} ||} | |jz} | dz|_t |_||_tj|| dzd|_ |rtjtj| |_ |jdtj| dtjtj| |_nd|_ d|_d|_|r|||_d|d zd z d|d zd z zdz|_tjtj|j$||_|jd t)|dnd|_d|_d|_tj,||_tj| ||_tj,||_y) akInitialize attention module. Args: dim: Input feature dimension. num_heads: Number of attention heads. qkv_bias: If True, add learnable bias to query, key, value projections. qkv_bias_separate: If True, use separate bias for q, k, v projections. attn_drop: Dropout rate for attention weights. proj_drop: Dropout rate for output projection. window_size: Window size for relative position bias. If None, no relative position bias. attn_head_dim: Dimension per attention head. If None, uses dim // num_heads. Ngr#F)biask_bias) persistentr"rrr4)super__init__r<scalerr:r>nnLinearqkv Parameterr'r-q_biasregister_bufferv_biasrDrr/relative_position_bias_tabler6r4Dropoutr?projr@) selfr;r<r=r>r?r@rrAhead_dim all_head_dim __class__s r5rGzAttention.__init__ds. ")#  $$H$..0 % (*!299S,"2? ,,u{{<'@ADK  5;;|+DQV W,,u{{<'@ADKDKDKDK *D *+k!n*Flinearr`reshaper<r+unbindr:rPr\scaled_dot_product_attentiontrainingr?prH transposesoftmaxrRr@) rSr]r^BNCrKr=qkv rel_pos_biasattns r5forwardzAttention.forwards''1a ;; ((1+Cyy$++t{{DKK!HIH%%hhqkxhhqxHkk!Q4>>26>>q!Q1M**Q-1a ??L00<#557 &2#/2E#EL$02 ..1a&.2mm$..**A DJJA B++D00<d4466".11<>$'DqA KK1  % %aA . IIaL NN1 r7)FFrbrbNNN)__name__ __module__ __qualname____doc__r'jitFinalbool__annotations__intfloatrr rGTensorr\rz __classcell__rVs@r5r9r9\s  %% "&+!!59+/8/8/8/ 8/ $ 8/  8/8/"%S/28/$C=8/t 35<< 366HU\\