arXiv:2601.08131v4 Announce Type: replace 
Abstract: Cross-layer reuse of early attention projections can improve optimization and data efficiency, but it creates a structural conflict: the first layer must simultaneously act as a stable, reusable anchor for all deeper layers and as an effective computational block. We demonstrate that this tension constrains the performance of internal-anchor designs. We propose ExoFormer, which resolves the conflict by learning exogenous anchor projections outside the sequential layer stack. We introduce a unified normalized mixing framework that mixes queries, keys, values, and gate logits using learnable coefficients (exploring coefficient granularities: elementwise, headwise, and scalar), and we show that normalizing anchor sources is key to stable reuse. ExoFormer variants consistently outperform their internal-anchor counterparts, and the dynamic variant yields 1.5x downstream accuracy points while matching validation loss using 1.5x fewer tokens than Gated Attention. We explain this efficacy via an Offloading Hypothesis: external anchors preserve essential token identity, allowing layers to specialize exclusively in feature transformation. We release code and models to facilitate future research.

أدت التطورات الأخيرة في آليات الانتباه إلى تقديم نموذج ExoFormer، الذي يستخدم إسقاطات مرجعية خارجية لتحسين التهيئة وكفاءة البيانات في هياكل التعلم العميق. تعالج هذه الطريقة الصراع الكامن في تصميمات المراسي الداخلية التقليدية، مما يسمح بتحسين الأداء في المهام اللاحقة.

Los avances recientes en los mecanismos de atención han llevado a la introducción de ExoFormer, un modelo que utiliza proyecciones de anclaje exógeno para mejorar la optimización y la eficiencia de los datos en arquitecturas de aprendizaje profundo. Este enfoque aborda el conflicto inherente en los diseños de anclaje interno tradicionales, permitiendo una mejor rendimiento en tareas posteriores.

Des avancées récentes dans les mécanismes d'attention ont conduit à l'introduction d'ExoFormer, un modèle qui utilise des projections d'ancrage exogènes pour améliorer l'optimisation et l'efficacité des données dans les architectures d'apprentissage profond. Cette approche résout le conflit inhérent dans les conceptions d'ancrage interne traditionnelles, permettant une meilleure performance dans les tâches en aval.

Recent advancements in attention mechanisms have led to the introduction of ExoFormer, a model that utilizes exogenous anchor projections to enhance optimization and data efficiency in deep learning architectures. This approach addresses the inherent conflict in traditional internal-anchor designs, allowing for improved performance in downstream tasks.