Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models

The recent publication titled 'Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models' presents significant findings on the internal mechanisms of multilingual large language models (LLMs). By introducing Language Attention Head Importance Scores (LAHIS), the study effectively identifies the importance of various attention heads in supporting multilingual capabilities. This method was applied to notable models such as Aya-23-8B, Llama-3.2-3B, and Mistral-7B-v0.1, revealing the presence of both language-specific and language-general heads. These findings are pivotal as they enhance cross-lingual attention transfer, helping to guide models toward appropriate language contexts and mitigate issues related to off-target language generation. The implications of this research extend beyond mere performance improvements; they also contribute to the interpretability of LLMs, making it easier to understand how these models process multiple l…