arXiv:2511.07498v2 Announce Type: replace 
Abstract: Large language models (LLMs) increasingly support multilingual understanding and generation. Meanwhile, efforts to interpret their internal mechanisms have emerged, offering insights to enhance multilingual performance. While multi-head self-attention (MHA) has proven critical in many areas, its role in multilingual capabilities remains underexplored. In this work, we study the contribution of MHA in supporting multilingual processing in LLMs. We propose Language Attention Head Importance Scores (LAHIS), an effective and efficient method that identifies attention head importance for multilingual capabilities via a single forward and backward pass through the LLM. Applying LAHIS to Aya-23-8B, Llama-3.2-3B, and Mistral-7B-v0.1, we reveal the existence of both language-specific and language-general heads. Language-specific heads enable cross-lingual attention transfer to guide the model toward target language contexts and mitigate off-target language generation issue, contributing to addressing challenges in multilingual LLMs. We also introduce a lightweight adaptation that learns a soft head mask to modulate attention outputs over language heads, requiring only 20 tunable parameters to improve XQuAD accuracy. Overall, our work enhances both the interpretability and multilingual capabilities of LLMs from the perspective of MHA.

أظهرت دراسة حديثة أهمية الانتباه متعدد الرؤوس (MHA) في نماذج اللغة متعددة اللغات (LLMs)، حيث تم تقديم طريقة تسمى درجات أهمية رؤوس الانتباه اللغوي (LAHIS) لتقييم أهمية رؤوس الانتباه. طبقت هذه الدراسة LAHIS على نماذج مثل Aya-23-8B وLlama-3.2-3B وMistral-7B-v0.1، وكشفت عن وجود رؤوس انتباه خاصة باللغة وأخرى عامة تسهل نقل الانتباه بين اللغات.

Un estudio reciente ha revelado la importancia de la atención multi-cabeza (MHA) en los modelos de lenguaje multilingües (LLMs), introduciendo un método llamado Puntuaciones de Importancia de Cabezas de Atención Lingüística (LAHIS) para evaluar la relevancia de las cabezas de atención. Esta investigación aplicó LAHIS a modelos como Aya-23-8B, Llama-3.2-3B y Mistral-7B-v0.1, descubriendo tanto cabezas específicas de idioma como generales que facilitan la transferencia de atención entre idiomas.

Une étude récente a révélé l'importance de l'attention multi-tête (MHA) dans les modèles de langage multilingues (LLMs), introduisant une méthode appelée Scores d'Importance des Têtes d'Attention Linguistique (LAHIS) pour évaluer la signification des têtes d'attention. Cette recherche a appliqué LAHIS à des modèles tels que Aya-23-8B, Llama-3.2-3B et Mistral-7B-v0.1, découvrant à la fois des têtes d'attention spécifiques à une langue et générales qui facilitent le transfert d'attention interlinguistique.

A recent study has revealed the importance of multi-head self-attention (MHA) in multilingual large language models (LLMs), introducing a method called Language Attention Head Importance Scores (LAHIS) to assess attention head significance. This research applied LAHIS to models such as Aya-23-8B, Llama-3.2-3B, and Mistral-7B-v0.1, uncovering both language-specific and language-general attention heads that facilitate cross-lingual attention transfer.

Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models

One More Thing in AI – Your Shortcut to AI Mastery

Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

OpenL Translator

Supametas.AI

Langfuse

Langtail

Ready to build your own newsroom?