arXiv:2605.27345v1 Announce Type: new 
Abstract: Reliable evaluation is essential for understanding large language model (LLM) performance, yet today's go-to metrics, namely token-overlap scores (e.g., ROUGE) and embedding-based measures (e.g., BERTScore), often misjudge semantic similarity of documents. Our study shows that both token-overlap metrics and embedding-based metrics routinely assign nearly identical scores to texts that directly contradict each other, thereby potentially masking fundamental errors. We introduce MATCHA, an automatic metric that jointly rewards semantic agreement with a reference and penalizes contradictions. MATCHA employs a dual-view perspective that measures (i) proximity to the gold text and (ii) distance from an adversarially generated counterfactual contradiction. In eight public benchmarks, MATCHA outperforms popular metrics, compared with human annotations on question-answering, image caption generation, natural language inference, summarization, and semantic textual similarity tasks. On the TruthfulQA dataset (i.e., a dataset without a training set, where no embedding-based metrics could locally train on), this improvement in terms of matching texts with a reference reaches 18.38% over ROUGE-L and 20.82% over BERTScore. Both quantitative comparison and qualitative human assessments confirm the efficacy and validity of MATCHA and uncover fundamental weaknesses in pre-existing metrics. Compared with 23 embedding models, including top state-of-the-art ones, used as a metric similar to BERTScore, MATCHA remains the most accurate in distinguishing correct from incorrect statements solely based on a reference. Our code and metric are publicly available (https://github.com/Siran-Li/MATCHA).

تم تقديم مقياس تقييم جديد يسمى MATCHA لتحسين موثوقية تقييم أداء نماذج اللغة الكبيرة (LLM)، حيث يعالج أوجه القصور في المقاييس الحالية مثل ROUGE وBERTScore التي غالبًا ما تحكم بشكل خاطئ على التشابه الدلالي. يقوم MATCHA بتقييم النصوص من خلال مكافأة الاتفاق الدلالي مع مرجع ومعاقبة التناقضات، مما يظهر أداءً متفوقًا عبر ثمانية معايير عامة.

Se ha introducido una nueva métrica de evaluación llamada MATCHA para mejorar la fiabilidad de la evaluación del rendimiento de los modelos de lenguaje de gran tamaño (LLM), abordando las deficiencias de métricas existentes como ROUGE y BERTScore que a menudo juzgan incorrectamente la similitud semántica. MATCHA evalúa los textos recompensando el acuerdo semántico con una referencia y penalizando las contradicciones, mostrando un rendimiento superior en ocho conjuntos de datos públicos.

Un nouveau métrique d'évaluation nommé MATCHA a été introduit pour améliorer la fiabilité de l'évaluation des performances des modèles de langage de grande taille (LLM), en abordant les lacunes des métriques existantes comme ROUGE et BERTScore qui jugent souvent mal la similarité sémantique. MATCHA évalue les textes en récompensant l'accord sémantique avec une référence tout en pénalisant les contradictions, montrant une performance supérieure dans huit ensembles de données publics.

A new evaluation metric named MATCHA has been introduced to enhance the reliability of assessing large language model (LLM) performance, addressing shortcomings in existing metrics like ROUGE and BERTScore that often misjudge semantic similarity. MATCHA evaluates texts by rewarding semantic agreement with a reference while penalizing contradictions, demonstrating superior performance across eight public benchmarks.