arXiv:2505.21501v3 Announce Type: replace 
Abstract: Vision Transformers (ViTs) have emerged as the dominant architecture for visual processing tasks, demonstrating excellent scalability with increased training data and model size. However, recent work has identified the emergence of artifact tokens in ViTs that are incongruous with local semantics. These anomalous tokens degrade ViT performance in tasks that require fine-grained localization or structural coherence. An effective mitigation of this issue is the addition of register tokens to ViTs, which implicitly "absorb" the artifact term during training. Given the availability of existing large-scale pre-trained ViTs, in this paper we seek add register tokens to existing models without needing to re-train from scratch, which is infeasible considering their size. Specifically, we propose Post Hoc Registers (PH-Reg), an efficient self-distillation method that integrates registers into an existing ViT without requiring additional labeled data and full retraining. PH-Reg initializes both teacher and student networks from the same pre-trained ViT. The teacher remains frozen and unmodified, while the student is augmented with randomly initialized register tokens. By applying test-time augmentation to the teacher's inputs, we generate denoised dense embeddings free of artifacts, which are then used to optimize only a small subset of unlocked student weights. We show that our approach can effectively reduce the number of artifact tokens, improving the segmentation and depth prediction of the student ViT under zero-shot and linear probing.

أصبحت محولات الرؤية (ViTs) هي الهيكل السائد لمهام معالجة الصور، حيث أظهرت قابلية رائعة للتوسع مع زيادة بيانات التدريب وحجم النموذج. ومع ذلك، حددت الأعمال الأخيرة ظهور رموز غير مرغوب فيها في ViTs تتعارض مع الدلالات المحلية. هذه الرموز الشاذة تؤثر سلبًا على أداء ViT في المهام التي تتطلب تحديدًا دقيقًا أو تماسكًا هيكليًا. يقدم هذا البحث إضافة رموز التسجيل إلى ViTs كوسيلة فعالة للتخفيف من هذه المشكلة، حيث يقترح استخدام رموز ما بعد التقييم (PH-Reg) كطريقة فعالة للاستخلاص الذاتي.

Los Vision Transformers (ViTs) se han convertido en la arquitectura dominante para tareas de procesamiento visual, demostrando una excelente escalabilidad con conjuntos de datos de entrenamiento y tamaños de modelo más grandes. Sin embargo, trabajos recientes han identificado la aparición de tokens de artefacto en los ViTs que son incongruentes con la semántica local. Estos tokens anómalos degradan el rendimiento de los ViTs en tareas que requieren localización precisa o coherencia estructural. Este artículo propone la adición de tokens de registro a los ViTs como una mitigación eficaz de este…

Les Vision Transformers (ViTs) sont devenus l'architecture dominante pour les tâches de traitement visuel, montrant une évolutivité remarquable avec des ensembles de données d'entraînement et des tailles de modèles plus importants. Cependant, des travaux récents ont révélé la présence de tokens d'artéfacts dans les ViTs qui sont en désaccord avec la sémantique locale, ce qui nuit aux performances dans les tâches nécessitant une localisation précise et une cohérence structurelle. Cet article propose d'ajouter des tokens d'enregistrement pour atténuer ce problème, en introduisant les Post Hoc Re…

Vision Transformers (ViTs) have become the leading architecture for visual processing tasks, showcasing remarkable scalability with larger training datasets and model sizes. However, recent findings have revealed the presence of artifact tokens in ViTs that conflict with local semantics, negatively impacting performance in tasks requiring precise localization and structural coherence. This paper introduces register tokens to mitigate this issue, proposing Post Hoc Registers (PH-Reg) as an efficient self-distillation method to integrate these tokens into existing ViTs without the need for retra…

Vision Transformers with Self-Distilled Registers

arXiv:2601.08499v1 Announce Type: new 
Abstract: Large models such as Vision Transformers (ViTs) have demonstrated remarkable superiority over smaller architectures like ResNet in few-shot classification, owing to their powerful representational capacity. However, fine-tuning such large models demands extensive GPU memory and prolonged training time, making them impractical for many real-world low-resource scenarios. To bridge this gap, we propose EfficientFSL, a query-only fine-tuning framework tailored specifically for few-shot classification with ViT, which achieves competitive performance while significantly reducing computational overhead. EfficientFSL fully leverages the knowledge embedded in the pre-trained model and its strong comprehension ability, achieving high classification accuracy with an extremely small number of tunable parameters. Specifically, we introduce a lightweight trainable Forward Block to synthesize task-specific queries that extract informative features from the intermediate representations of the pre-trained model in a query-only manner. We further propose a Combine Block to fuse multi-layer outputs, enhancing the depth and robustness of feature representations. Finally, a Support-Query Attention Block mitigates distribution shift by adjusting prototypes to align with the query set distribution. With minimal trainable parameters, EfficientFSL achieves state-of-the-art performance on four in-domain few-shot datasets and six cross-domain datasets, demonstrating its effectiveness in real-world applications.

يقدم EfficientFSL إطارًا للتعديل يعتمد فقط على الاستفسارات لتحسين تصنيف القليل من العينات باستخدام المحولات البصرية (ViTs)، مما يعزز الأداء مع تقليل كبير في المتطلبات الحاسوبية. تستفيد هذه الطريقة من قدرات النموذج المدرب مسبقًا، محققة دقة عالية مع عدد قليل من المعلمات القابلة للتعديل.

EfficientFSL presenta un marco de ajuste solo de consultas para Transformadores de Visión (ViTs), mejorando la clasificación de pocos disparos mientras reduce significativamente las demandas computacionales. Este enfoque aprovecha las capacidades del modelo preentrenado, logrando alta precisión con un número mínimo de parámetros.

EfficientFSL introduit un cadre de réglage uniquement basé sur les requêtes pour les Transformateurs de Vision (ViTs), améliorant la classification à faible échantillon tout en réduisant considérablement les exigences computationnelles. Cette approche exploite les capacités du modèle pré-entraîné, atteignant une haute précision avec un nombre minimal de paramètres.

EfficientFSL introduces a query-only fine-tuning framework for Vision Transformers (ViTs), enhancing few-shot classification while significantly reducing computational demands. This approach leverages the pre-trained model's capabilities, achieving high accuracy with minimal parameters.

EfficientFSL: Enhancing Few-Shot Classification via Query-Only Tuning in Vision Transformers

arXiv:2601.08602v1 Announce Type: new 
Abstract: Vision modeling has advanced rapidly with Transformers, whose attention mechanisms capture visual dependencies but lack a principled account of how semantic information propagates spatially. We revisit this problem from a wave-based perspective: feature maps are treated as spatial signals whose evolution over an internal propagation time (aligned with network depth) is governed by an underdamped wave equation. In this formulation, spatial frequency-from low-frequency global layout to high-frequency edges and textures-is modeled explicitly, and its interaction with propagation time is controlled rather than implicitly fixed. We derive a closed-form, frequency-time decoupled solution and implement it as the Wave Propagation Operator (WPO), a lightweight module that models global interactions in O(N log N) time-far lower than attention. Building on WPO, we propose a family of WaveFormer models as drop-in replacements for standard ViTs and CNNs, achieving competitive accuracy across image classification, object detection, and semantic segmentation, while delivering up to 1.6x higher throughput and 30% fewer FLOPs than attention-based alternatives. Furthermore, our results demonstrate that wave propagation introduces a complementary modeling bias to heat-based methods, effectively capturing both global coherence and high-frequency details essential for rich visual semantics. Codes are available at: https://github.com/ZishanShu/WaveFormer.

تقدم دراسة جديدة WaveFormer، وهو نهج لنمذجة الرؤية يستخدم معادلة موجية للتحكم في تطور خرائط الميزات بمرور الوقت، مما يعزز نمذجة الترددات المكانية والتفاعلات في البيانات المرئية. توفر هذه الطريقة حلاً مغلقًا يتم تنفيذه كعامل انتشار الموجات (WPO)، الذي يعمل بكفاءة أكبر من آليات الانتباه التقليدية.

Un nuevo estudio presenta WaveFormer, un enfoque de modelado visual que utiliza una ecuación de onda para gobernar la evolución de los mapas de características a lo largo del tiempo, mejorando así la modelación de frecuencias espaciales e interacciones en los datos visuales. Este método ofrece una solución en forma cerrada implementada como el Operador de Propagación de Onda (WPO), que opera de manera más eficiente que los mecanismos de atención tradicionales.

Une nouvelle étude présente WaveFormer, une approche de modélisation visuelle qui utilise une équation d'onde pour gouverner l'évolution des cartes de caractéristiques dans le temps, améliorant ainsi la modélisation des fréquences spatiales et des interactions dans les données visuelles. Cette méthode offre une solution sous forme fermée mise en œuvre comme l'Opérateur de Propagation d'Onde (WPO), qui fonctionne plus efficacement que les mécanismes d'attention traditionnels.

A new study introduces WaveFormer, a vision modeling approach that utilizes a wave equation to govern the evolution of feature maps over time, enhancing the modeling of spatial frequencies and interactions in visual data. This method offers a closed-form solution implemented as the Wave Propagation Operator (WPO), which operates more efficiently than traditional attention mechanisms.

Vision Transformers with Self-Distilled Registers

Was this article worth reading? Share it

Airparser

Humanize AI

The Visualizer

Videotok

GPTHumanizer

Velory

Ready to build your own newsroom?