arXiv:2512.05858v1 Announce Type: new 
Abstract: This is the fourth in a series of short reports that help business, education, and policy leaders understand the technical details of working with AI through rigorous testing. Here, we ask whether assigning personas to models improves performance on difficult objective multiple-choice questions. We study both domain-specific expert personas and low-knowledge personas, evaluating six models on GPQA Diamond (Rein et al. 2024) and MMLU-Pro (Wang et al. 2024), graduate-level questions spanning science, engineering, and law.
  We tested three approaches:
  -In-Domain Experts: Assigning the model an expert persona ("you are a physics expert") matched to the problem type (physics problems) had no significant impact on performance (with the exception of the Gemini 2.0 Flash model).
  -Off-Domain Experts (Domain-Mismatched): Assigning the model an expert persona ("you are a physics expert") not matched to the problem type (law problems) resulted in marginal differences.
  -Low-Knowledge Personas: We assigned the model negative capability personas (layperson, young child, toddler), which were generally harmful to benchmark accuracy.
  Across both benchmarks, persona prompts generally did not improve accuracy relative to a no-persona baseline. Expert personas showed no consistent benefit across models, with few exceptions. Domain-mismatched expert personas sometimes degraded performance. Low-knowledge personas often reduced accuracy. These results are about the accuracy of answers only; personas may serve other purposes (such as altering the tone of outputs), beyond improving factual performance.

التقرير الرابع في سلسلة تقارير Prompting Science يستكشف ما إذا كان تعيين شخصيات خبراء لنماذج الذكاء الاصطناعي يحسن أدائها في الأسئلة متعددة الخيارات الصعبة. قامت الدراسة بتقييم ستة نماذج، بما في ذلك Gemini 2.0 Flash، باستخدام إطارين للتقييم، GPQA Diamond وMMLU-Pro، ولم تجد أي تحسين كبير في الدقة مع شخصيات الخبراء، باستثناء نموذج واحد.

El cuarto informe de la serie Prompting Science investiga si asignar personas expertas a los modelos de IA mejora su rendimiento en preguntas de opción múltiple desafiantes. El estudio evaluó seis modelos, incluido Gemini 2.0 Flash, utilizando dos marcos de evaluación, GPQA Diamond y MMLU-Pro, y no encontró una mejora significativa en la precisión con las personas expertas, excepto en un modelo.

Le quatrième rapport de la série Prompting Science examine si l'attribution de personas d'experts aux modèles d'IA améliore leur performance sur des questions à choix multiples difficiles. L'étude a évalué six modèles, dont Gemini 2.0 Flash, en utilisant deux cadres d'évaluation, GPQA Diamond et MMLU-Pro, et n'a trouvé aucune amélioration significative de la précision avec les personas d'experts, sauf pour un modèle.

The fourth report in the Prompting Science series investigates whether assigning expert personas to AI models enhances their performance on challenging multiple-choice questions. The study evaluated six models, including Gemini 2.0 Flash, using two assessment frameworks, GPQA Diamond and MMLU-Pro, and found no significant improvement in accuracy with expert personas, except for one model.

Prompting Science Report 4: Playing Pretend: Expert Personas Don't Improve Factual Accuracy

One More Thing in AI – Your Shortcut to AI Mastery

Prompting Science Report 4: Playing Pretend: Expert Personas Don't Improve Factual Accuracy

Was this article worth reading? Share it

One More Thing in AI

Humanize AI

Guidejar-4eb95b

TheQuizMaster

Synthx

Personos

Ready to build your own newsroom?