Disentangling Language Roles in Multilingual LLM Task Execution

A new benchmark called MTM-Bench has been introduced to evaluate multilingual large language models (LLMs) in task execution, focusing on the roles of instruction, content, and response languages. This benchmark encompasses 27 language triplets across English, Spanish, and Chinese, with 2,430 instances per model to assess various metrics including semantic correctness and language adherence.

The development of MTM-Bench is significant as it aims to provide a controlled environment for evaluating LLMs, addressing the complexities of multilingual task execution where language roles often overlap. This could enhance the reliability and effectiveness of LLMs in diverse linguistic contexts.

This initiative reflects a broader trend in AI research towards improving multilingual capabilities and addressing biases inherent in LLMs, particularly the English-centric focus observed in many models. The ongoing exploration of multilingual frameworks and evaluation metrics is crucial for advancing the field and ensuring equitable performance across languages.