arXiv:2511.10664v1 Announce Type: new 
Abstract: Large language models (LLMs) have achieved impressive results in high-resource languages like English, yet their effectiveness in low-resource and morphologically rich languages remains underexplored. In this paper, we present a comprehensive evaluation of seven cutting-edge LLMs -- including GPT-4o, GPT-4, Claude~3.5~Sonnet, LLaMA~3.1, Mistral~Large~2, LLaMA-2~Chat~13B, and Mistral~7B~Instruct -- on a new cross-lingual benchmark covering \textbf{Cantonese, Japanese, and Turkish}. Our benchmark spans four diverse tasks: open-domain question answering, document summarization, English-to-X translation, and culturally grounded dialogue. We combine \textbf{human evaluations} (rating fluency, factual accuracy, and cultural appropriateness) with automated metrics (e.g., BLEU, ROUGE) to assess model performance.
  Our results reveal that while the largest proprietary models (GPT-4o, GPT-4, Claude~3.5) generally lead across languages and tasks, significant gaps persist in culturally nuanced understanding and morphological generalization. Notably, GPT-4o demonstrates robust multilingual performance even on cross-lingual tasks, and Claude~3.5~Sonnet achieves competitive accuracy on knowledge and reasoning benchmarks. However, all models struggle to some extent with the unique linguistic challenges of each language, such as Turkish agglutinative morphology and Cantonese colloquialisms. Smaller open-source models (LLaMA-2~13B, Mistral~7B) lag substantially in fluency and accuracy, highlighting the resource disparity. We provide detailed quantitative results, qualitative error analysis, and discuss implications for developing more culturally aware and linguistically generalizable LLMs. Our benchmark and evaluation data are released to foster reproducibility and further research.

تقيّم دراسة حديثة أداء سبعة نماذج متقدمة للغة (LLMs) على لغات ذات موارد منخفضة وغنية بالمورفولوجيا، تحديدًا الكانتونية واليابانية والتركية. تبرز الأبحاث فعالية النماذج في مهام مثل الإجابة على الأسئلة المفتوحة، تلخيص الوثائق، الترجمة، والحوار المستند إلى الثقافة. على الرغم من النتائج المثيرة للإعجاب في اللغات ذات الموارد العالية، تشير الدراسة إلى أن فعالية LLMs في هذه اللغات الأقل دراسة لا تزال غير مستكشفة.

Un estudio reciente evalúa el rendimiento de siete modelos de lenguaje avanzados (LLMs) en lenguas de bajos recursos y morfológicamente ricas, específicamente cantonés, japonés y turco. La investigación destaca la efectividad de los modelos en tareas como respuesta a preguntas de dominio abierto, resumen de documentos, traducción y diálogo culturalmente fundamentado. A pesar de los resultados impresionantes en lenguas de altos recursos, el estudio indica que la efectividad de los LLMs en estas lenguas menos estudiadas sigue siendo poco explorada.

Une étude récente évalue la performance de sept modèles de langage avancés (LLMs) sur des langues à faibles ressources et morphologiquement riches, notamment le cantonais, le japonais et le turc. La recherche met en évidence l'efficacité des modèles dans des tâches telles que la réponse à des questions en domaine ouvert, la synthèse de documents, la traduction et le dialogue culturellement ancré. Malgré des résultats impressionnants dans les langues à fortes ressources, l'étude indique que l'efficacité des LLMs dans ces langues moins étudiées reste sous-explorée.

A recent study evaluates the performance of seven advanced large language models (LLMs) on low-resource and morphologically rich languages, specifically Cantonese, Japanese, and Turkish. The research highlights the models' effectiveness in tasks such as open-domain question answering, document summarization, translation, and culturally grounded dialogue. Despite impressive results in high-resource languages, the study indicates that the effectiveness of LLMs in these less-studied languages remains underexplored.

Evaluating Modern Large Language Models on Low-Resource and Morphologically Rich Languages:A Cross-Lingual Benchmark Across Cantonese, Japanese, and Turkish

Evaluating Modern Large Language Models on Low-Resource and Morphologically Rich Languages:A Cross-Lingual Benchmark Across Cantonese, Japanese, and Turkish

Was this article worth reading? Share it

Airparser

ChatOne

Langtail

ZeroGPT.org

OpenL Translator

Sellm

Ready to build your own newsroom?