arXiv:2602.07842v2 Announce Type: replace 
Abstract: Confidence calibration is essential for making large language models (LLMs) reliable, yet existing training-free methods have been primarily studied under single-answer question answering. In this paper, we show that these methods break down in the presence of multiple valid answers, where disagreement among equally correct responses leads to systematic underestimation of confidence. To enable a systematic study of this phenomenon, we introduce MACE, a benchmark of 12,000 factual questions spanning six domains with varying numbers of correct answers. Experiments across 15 representative calibration methods and four LLM families (7B-72B) reveal that while accuracy increases with answer cardinality, estimated confidence consistently decreases, causing severe miscalibration for questions with mixed answer counts. To address this issue, we propose Semantic Confidence Aggregation (SCA), which aggregates confidence over multiple high-probability sampled responses. SCA achieves state-of-the-art calibration performance under mixed-answer settings while preserving strong calibration on single-answer questions.

أظهرت دراسة حديثة تقديم MACE، وهو معيار يتكون من 12,000 سؤال واقعي عبر ستة مجالات، لتقييم ضبط ثقة نماذج اللغة الكبيرة (LLMs) عند مواجهة إجابات صحيحة متعددة. تكشف الأبحاث أن طرق الضبط الحالية التي لا تتطلب تدريبًا غالبًا ما تقلل من مستويات الثقة بسبب الاختلافات بين الإجابات الصحيحة على حد سواء.

Un estudio reciente ha presentado MACE, un conjunto de referencia que consta de 12,000 preguntas fácticas en seis dominios, para evaluar la calibración de la confianza de los modelos de lenguaje grandes (LLMs) cuando se enfrentan a múltiples respuestas correctas. La investigación revela que los métodos de calibración sin entrenamiento existentes a menudo subestiman los niveles de confianza debido a desacuerdos entre respuestas igualmente válidas.

Une étude récente a introduit MACE, une référence composée de 12 000 questions factuelles réparties sur six domaines, pour évaluer la calibration de la confiance des grands modèles de langage (LLMs) face à des réponses multiples correctes. La recherche révèle que les méthodes de calibration sans formation existantes sous-estiment souvent les niveaux de confiance en raison de désaccords entre des réponses également valides.

A recent study has introduced MACE, a benchmark consisting of 12,000 factual questions across six domains, to evaluate the confidence calibration of large language models (LLMs) when faced with multiple correct answers. The research reveals that existing training-free calibration methods often misestimate confidence levels due to disagreements among equally valid responses.