arXiv:2511.01292v1 Announce Type: cross 
Abstract: Pretrained Transformers excel at in-context learning (ICL), inferring new tasks from only a handful of examples. Yet, their ICL performance can degrade sharply under distribution shift between pretraining and test data, a regime increasingly common in real-world deployments. While recent empirical work hints that adjusting the attention temperature in the softmax can enhance Transformer performance, the attention temperature's role in ICL under distribution shift remains unexplored. This paper provides the first theoretical and empirical study of attention temperature for ICL under distribution shift. Using a simplified but expressive "linearized softmax" framework, we derive closed-form generalization error expressions and prove that shifts in input covariance or label noise substantially impair ICL, but that an optimal attention temperature exists which minimizes this error. We then validate our predictions through extensive simulations on linear regression tasks and large-scale experiments with GPT-2 and LLaMA2-7B on question-answering benchmarks. Our results establish attention temperature as a principled and powerful mechanism for improving the robustness of ICL in pretrained Transformers, advancing theoretical understanding and providing actionable guidance for selecting attention temperature in practice.

تسلط الأبحاث الحديثة الضوء على كيفية تحسين درجة حرارة الانتباه في نماذج Transformers لتعلم السياق، خاصة عند مواجهة تغيرات التوزيع بين التدريب والاختبار. هذه الخطوة مهمة لأنها تعالج تحديًا شائعًا في التطبيقات الواقعية، مما يضمن أداء هذه النماذج بشكل مثالي حتى في ظروف متغيرة.

Investigaciones recientes destacan cómo ajustar la temperatura de atención en los Transformers puede mejorar significativamente el aprendizaje en contexto, especialmente cuando se enfrenta a cambios de distribución entre el preentrenamiento y la prueba. Este avance es crucial, ya que aborda un desafío común en aplicaciones del mundo real, asegurando que estos modelos funcionen de manera óptima incluso en condiciones variables.

Des recherches récentes soulignent comment l'ajustement de la température d'attention dans les Transformers peut améliorer considérablement l'apprentissage en contexte, en particulier face à des changements de distribution entre l'entraînement et le test. Cette avancée est cruciale car elle répond à un défi courant dans les applications réelles, garantissant que ces modèles fonctionnent de manière optimale même dans des conditions variées.

Recent research highlights how adjusting the attention temperature in Transformers can significantly improve in-context learning, especially when faced with distribution shifts between pretraining and testing. This advancement is crucial as it addresses a common challenge in real-world applications, ensuring that these models perform optimally even in varying conditions.

Optimal Attention Temperature Enhances In-Context Learning under Distribution Shift

arXiv:2507.06560v3 Announce Type: replace-cross 
Abstract: Recent success in contrastive learning has sparked growing interest in more effectively leveraging multiple augmented views of data. While prior methods incorporate multiple views at the loss or feature level, they primarily capture pairwise relationships and fail to model the joint structure across all views. In this work, we propose a divergence-based similarity function (DSF) that explicitly captures the joint structure by representing each set of augmented views as a distribution and measuring similarity as the divergence between distributions. Extensive experiments demonstrate that DSF consistently improves performance across diverse tasks, including kNN classification, linear evaluation, transfer learning, and distribution shift, while also achieving greater efficiency than other multi-view methods. Furthermore, we establish a connection between DSF and cosine similarity, and demonstrate that, unlike cosine similarity, DSF operates effectively without the need for tuning a temperature hyperparameter.

تم اقتراح دالة تشابه جديدة تعتمد على التباين (DSF) للتعلم المتباين متعدد المناظر، بهدف تحسين فعالية استخدام عدة مناظر معززة للبيانات. تلتقط هذه الطريقة الهيكل المشترك للمناظر المعززة من خلال تمثيلها كإحصائيات وقياس التشابه من خلال التباين، مما يظهر تحسينًا في الأداء عبر مهام متنوعة مثل تصنيف kNN والتعلم الانتقالي.

Se ha propuesto una nueva función de similitud basada en la divergencia (DSF) para el aprendizaje contrastivo de múltiples vistas, con el objetivo de mejorar la efectividad de la utilización de múltiples vistas aumentadas de datos. Este método captura la estructura conjunta de las vistas aumentadas al representarlas como distribuciones y medir la similitud a través de la divergencia, demostrando una mejora en el rendimiento en diversas tareas como la clasificación kNN y el aprendizaje por transferencia.

Une nouvelle fonction de similarité basée sur la divergence (DSF) a été proposée pour l'apprentissage contrastif multi-vues, visant à améliorer l'efficacité de l'utilisation de plusieurs vues augmentées des données. Cette méthode capture la structure conjointe des vues augmentées en les représentant comme des distributions et en mesurant la similarité par la divergence, montrant ainsi une amélioration des performances dans diverses tâches telles que la classification kNN et l'apprentissage par transfert.

A new divergence-based similarity function (DSF) has been proposed for multi-view contrastive learning, aiming to enhance the effectiveness of utilizing multiple augmented views of data. This method captures the joint structure of augmented views by representing them as distributions and measuring similarity through divergence, demonstrating improved performance across various tasks such as kNN classification and transfer learning.

Divergence-Based Similarity Function for Multi-View Contrastive Learning

One More Thing in AI – Your Shortcut to AI Mastery

Optimal Attention Temperature Enhances In-Context Learning under Distribution Shift

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

Hypertune

The Visualizer

Cont3xt.dev

Attentive AI

Ready to build your own newsroom?