arXiv:2511.11683v1 Announce Type: new 
Abstract: Training and deploying multiple vision transformer (ViT) models for different resource constraints is costly and inefficient. To address this, we propose transforming a pre-trained ViT into a stratified knowledge-density super-network, where knowledge is hierarchically organized across weights. This enables flexible extraction of sub-networks that retain maximal knowledge for varying model sizes. We introduce \textbf{W}eighted \textbf{P}CA for \textbf{A}ttention \textbf{C}ontraction (WPAC), which concentrates knowledge into a compact set of critical weights. WPAC applies token-wise weighted principal component analysis to intermediate features and injects the resulting transformation and inverse matrices into adjacent layers, preserving the original network function while enhancing knowledge compactness. To further promote stratified knowledge organization, we propose \textbf{P}rogressive \textbf{I}mportance-\textbf{A}ware \textbf{D}ropout (PIAD). PIAD progressively evaluates the importance of weight groups, updates an importance-aware dropout list, and trains the super-network under this dropout regime to promote knowledge stratification. Experiments demonstrate that WPAC outperforms existing pruning criteria in knowledge concentration, and the combination with PIAD offers a strong alternative to state-of-the-art model compression and model expansion methods.

يقدم المقال نهجًا مبتكرًا لتحسين نماذج المحولات البصرية (ViT) من خلال إنشاء شبكة فائقة كثافة المعرفة ذات طبقات. تنظم هذه الطريقة المعرفة بشكل هرمي عبر الأوزان، مما يسمح بالاستخراج المرن للشبكات الفرعية التي تحتفظ بالمعرفة الأساسية لأحجام النماذج المختلفة. تم تقديم PCA الموزونة لتقليص الانتباه (WPAC) لتحسين كثافة المعرفة مع الحفاظ على وظيفة الشبكة الأصلية، مما يعالج عدم الكفاءة في تدريب نماذج ViT متعددة تحت قيود موارد مختلفة.

El artículo presenta un enfoque novedoso para optimizar los modelos de transformadores de visión (ViT) mediante la creación de una superred de densidad de conocimiento estratificada. Este método organiza el conocimiento de manera jerárquica a través de los pesos, permitiendo la extracción flexible de subredes que mantienen el conocimiento esencial para varios tamaños de modelo. La introducción de PCA Ponderada para la Contracción de Atención (WPAC) mejora la compacidad del conocimiento mientras preserva la función original de la red, abordando las ineficiencias del entrenamiento de múltiples m…

L'article présente une approche novatrice pour optimiser les modèles de transformateurs de vision (ViT) en créant un super-réseau stratifié de densité de connaissances. Cette méthode organise les connaissances de manière hiérarchique à travers les poids, permettant une extraction flexible de sous-réseaux qui conservent les connaissances essentielles pour différentes tailles de modèles. L'introduction de la PCA pondérée pour la contraction d'attention (WPAC) améliore la compacité des connaissances tout en préservant la fonction originale du réseau, répondant ainsi aux inefficacités de l'entraîn…

The article presents a novel approach to optimizing vision transformer (ViT) models by creating a stratified knowledge-density super-network. This method organizes knowledge hierarchically across weights, allowing for flexible extraction of sub-networks that maintain essential knowledge for various model sizes. The introduction of Weighted PCA for Attention Contraction (WPAC) enhances knowledge compactness while preserving the original network function, addressing the inefficiencies of training multiple ViT models under different resource constraints.

Stratified Knowledge-Density Super-Network for Scalable Vision Transformers

arXiv:2601.08499v1 Announce Type: new 
Abstract: Large models such as Vision Transformers (ViTs) have demonstrated remarkable superiority over smaller architectures like ResNet in few-shot classification, owing to their powerful representational capacity. However, fine-tuning such large models demands extensive GPU memory and prolonged training time, making them impractical for many real-world low-resource scenarios. To bridge this gap, we propose EfficientFSL, a query-only fine-tuning framework tailored specifically for few-shot classification with ViT, which achieves competitive performance while significantly reducing computational overhead. EfficientFSL fully leverages the knowledge embedded in the pre-trained model and its strong comprehension ability, achieving high classification accuracy with an extremely small number of tunable parameters. Specifically, we introduce a lightweight trainable Forward Block to synthesize task-specific queries that extract informative features from the intermediate representations of the pre-trained model in a query-only manner. We further propose a Combine Block to fuse multi-layer outputs, enhancing the depth and robustness of feature representations. Finally, a Support-Query Attention Block mitigates distribution shift by adjusting prototypes to align with the query set distribution. With minimal trainable parameters, EfficientFSL achieves state-of-the-art performance on four in-domain few-shot datasets and six cross-domain datasets, demonstrating its effectiveness in real-world applications.

يقدم EfficientFSL إطارًا للتعديل يعتمد فقط على الاستفسارات لتحسين تصنيف القليل من العينات باستخدام المحولات البصرية (ViTs)، مما يعزز الأداء مع تقليل كبير في المتطلبات الحاسوبية. تستفيد هذه الطريقة من قدرات النموذج المدرب مسبقًا، محققة دقة عالية مع عدد قليل من المعلمات القابلة للتعديل.

EfficientFSL presenta un marco de ajuste solo de consultas para Transformadores de Visión (ViTs), mejorando la clasificación de pocos disparos mientras reduce significativamente las demandas computacionales. Este enfoque aprovecha las capacidades del modelo preentrenado, logrando alta precisión con un número mínimo de parámetros.

EfficientFSL introduit un cadre de réglage uniquement basé sur les requêtes pour les Transformateurs de Vision (ViTs), améliorant la classification à faible échantillon tout en réduisant considérablement les exigences computationnelles. Cette approche exploite les capacités du modèle pré-entraîné, atteignant une haute précision avec un nombre minimal de paramètres.

EfficientFSL introduces a query-only fine-tuning framework for Vision Transformers (ViTs), enhancing few-shot classification while significantly reducing computational demands. This approach leverages the pre-trained model's capabilities, achieving high accuracy with minimal parameters.

EfficientFSL: Enhancing Few-Shot Classification via Query-Only Tuning in Vision Transformers

One More Thing in AI – Your Shortcut to AI Mastery

Stratified Knowledge-Density Super-Network for Scalable Vision Transformers

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

The Visualizer

Https

Octofy

FETCH HIVE

Ready to build your own newsroom?