arXiv:2506.00250v4 Announce Type: replace 
Abstract: Large Language Models (LLMs) have achieved remarkable performance on a wide range of Natural Language Processing (NLP) benchmarks, often surpassing human-level accuracy. However, their reliability in high-stakes domains such as medicine, particularly in low-resource languages, remains underexplored. In this work, we introduce PersianMedQA, a large-scale dataset of 20,785 expert-validated multiple-choice Persian medical questions from 14 years of Iranian national medical exams, spanning 23 medical specialties and designed to evaluate LLMs in both Persian and English. We benchmark 41 state-of-the-art models, including general-purpose, Persian, and medical LLMs, in zero-shot and chain-of-thought (CoT) settings. Our results show that closed-weight general models (e.g., GPT-4.1) consistently outperform all other categories, achieving 83.09% accuracy in Persian and 80.7% in English, while Persian LLMs such as Dorna underperform significantly (e.g., 34.9% in Persian), often struggling with both instruction-following and domain reasoning. We also analyze the impact of translation, showing that while English performance is generally higher, 3-10% of questions can only be answered correctly in Persian due to cultural and clinical contextual cues that are lost in translation. Finally, we demonstrate that model size alone is insufficient for robust performance without strong domain or language adaptation. PersianMedQA provides a foundation for evaluating bilingual and culturally grounded medical reasoning in LLMs. The dataset, along with a bilingual medical dictionary, is available: https://huggingface.co/datasets/MohammadJRanjbar/PersianMedQA .

يمثل إدخال PersianMedQA تقدمًا كبيرًا في تقييم نماذج اللغة الكبيرة (LLMs) في المجال الطبي، حيث يتضمن مجموعة بيانات تضم 20,785 سؤالًا طبيًا فارسيًا تم التحقق منها من قبل خبراء، مستمدة من 14 عامًا من الامتحانات الطبية الوطنية الإيرانية عبر 23 تخصصًا. يهدف هذا المعيار إلى تقييم أداء نماذج LLMs المختلفة، بما في ذلك GPT-4.1 وDorna، في سياقات فارسية وإنجليزية.

La introducción de PersianMedQA marca un avance significativo en la evaluación de grandes modelos de lenguaje (LLMs) en el campo médico, presentando un conjunto de datos de 20,785 preguntas médicas persas validadas por expertos, derivadas de 14 años de exámenes médicos nacionales iraníes en 23 especialidades. Este benchmark tiene como objetivo evaluar el rendimiento de varios LLMs, incluidos GPT-4.1 y Dorna, en contextos persa e inglés.

L'introduction de PersianMedQA représente une avancée significative dans l'évaluation des grands modèles de langage (LLMs) dans le domaine médical, avec un ensemble de données de 20 785 questions médicales persanes validées par des experts, dérivées de 14 ans d'examens médicaux nationaux iraniens couvrant 23 spécialités. Ce benchmark vise à évaluer la performance de divers LLMs, y compris GPT-4.1 et Dorna, dans des contextes persan et anglais.

The introduction of PersianMedQA marks a significant advancement in evaluating large language models (LLMs) in the medical field, featuring a dataset of 20,785 expert-validated Persian medical questions derived from 14 years of Iranian national medical exams across 23 specialties. This benchmark aims to assess the performance of various LLMs, including GPT-4.1 and Dorna, in both Persian and English contexts.

PersianMedQA: Evaluating Large Language Models on a Persian-English Bilingual Medical Question Answering Benchmark

Was this article worth reading? Share it

Ready to build your own newsroom?