World PulseNow
Newsletter

Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QA

arXiv — cs.CL•Wednesday, November 12, 2025 at 5:00:00 AM
PositiveArtificial Intelligence
The emergence of Large Language Models (LLMs) has transformed the landscape of AI, particularly in generating human-like conversations. However, traditional evaluation metrics such as EM and F1 have proven inadequate for assessing the nuanced outputs of these models. In response, researchers have introduced a reference-guided verdict method that leverages multiple LLMs as judges to enhance the evaluation process. Experiments conducted on free-form question-answering tasks reveal that this approach significantly improves the reliability and accuracy of evaluations. The results indicate a strong correlation with human evaluations, establishing the proposed method as a credible alternative to existing metrics. This advancement is crucial as it not only addresses the limitations of conventional evaluation methods but also paves the way for more sophisticated AI assessment techniques, ensuring that the capabilities of LLMs are accurately measured and understood.
— via World Pulse Now AI Editorial System
Read Original

Was this article worth reading? Share it

Recommended Readings
Silenced Biases: The Dark Side LLMs Learned to Refuse
arXiv — stat.ML16 hours ago
Silenced Biases: The Dark Side LLMs Learned to Refuse
NegativeArtificial Intelligence
Safety-aligned large language models (LLMs) are increasingly used in sensitive applications where fairness is crucial. Evaluating their fairness is complex, often relying on standard question-answer schemes that may misinterpret refusal responses as indicators of fairness. This paper introduces the concept of silenced biases, which are unfair preferences hidden within the models' latent space, masked by safety-alignment. Previous methods have limitations, prompting the need for a new approach to assess these biases effectively.
Read full article
via arXiv — stat.ML
Fair In-Context Learning via Latent Concept Variables
arXiv — cs.LG16 hours ago
Fair In-Context Learning via Latent Concept Variables
PositiveArtificial Intelligence
The paper titled 'Fair In-Context Learning via Latent Concept Variables' explores the in-context learning (ICL) capabilities of large language models (LLMs) and their potential biases when applied to tabular data. It emphasizes an optimal demonstration selection method that leverages latent concept variables to enhance task adaptation while promoting fairness. The study introduces data augmentation strategies aimed at minimizing correlations between sensitive variables and predictive outcomes, ultimately striving for equitable predictions.
Read full article
via arXiv — cs.LG
Multimodal Peer Review Simulation with Actionable To-Do Recommendations for Community-Aware Manuscript Revisions
arXiv — cs.CL2 days ago
Multimodal Peer Review Simulation with Actionable To-Do Recommendations for Community-Aware Manuscript Revisions
PositiveArtificial Intelligence
A new interactive web-based system for multimodal peer review simulation has been introduced, aimed at enhancing manuscript revisions prior to submission. This system leverages large language models (LLMs) to integrate textual and visual information, improving the quality of reviews through retrieval-augmented generation (RAG) based on OpenReview data. It converts generated reviews into actionable to-do lists, providing structured guidance for authors and seamlessly integrating with existing academic writing platforms.
Read full article
via arXiv — cs.CL
Modeling and Predicting Multi-Turn Answer Instability in Large Language Models
arXiv — cs.CL2 days ago
Modeling and Predicting Multi-Turn Answer Instability in Large Language Models
NeutralArtificial Intelligence
The paper titled 'Modeling and Predicting Multi-Turn Answer Instability in Large Language Models' discusses the evaluation of large language models (LLMs) in terms of their robustness during user interactions. The study employs multi-turn follow-up prompts to assess changes in model answers and accuracy dynamics using Markov chains. Results indicate vulnerabilities in LLMs, with a 10% accuracy drop for Gemini 1.5 Flash after a 'Think again' prompt over nine turns, and a 7.5% drop for Claude 3.5 Haiku with a reworded question. The findings suggest that accuracy can be modeled over time.
Read full article
via arXiv — cs.CL
Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Models
arXiv — cs.CL2 days ago
Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Models
PositiveArtificial Intelligence
The paper titled 'Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Models' introduces a method to enhance the efficiency of Mixture-of-Experts (MoE) Large Language Models (LLMs). The authors propose a pre-attention expert prediction technique that improves accuracy and reduces computational overhead by utilizing activations before the attention block. This approach aims to optimize expert prefetching, achieving about a 15% improvement in accuracy over existing methods.
Read full article
via arXiv — cs.CL
From Fact to Judgment: Investigating the Impact of Task Framing on LLM Conviction in Dialogue Systems
arXiv — cs.CL2 days ago
From Fact to Judgment: Investigating the Impact of Task Framing on LLM Conviction in Dialogue Systems
NeutralArtificial Intelligence
The article investigates the impact of task framing on the conviction of large language models (LLMs) in dialogue systems. It explores how LLMs assess tasks requiring social judgment, contrasting their performance on factual queries with conversational judgment tasks. The study reveals that reframing a task can significantly alter an LLM's judgment, particularly under conversational pressure, highlighting the complexities of LLM decision-making in social contexts.
Read full article
via arXiv — cs.CL
Expert-Guided Prompting and Retrieval-Augmented Generation for Emergency Medical Service Question Answering
arXiv — cs.CL2 days ago
Expert-Guided Prompting and Retrieval-Augmented Generation for Emergency Medical Service Question Answering
PositiveArtificial Intelligence
Large language models (LLMs) have shown potential in medical question answering but often lack the domain-specific expertise required in emergency medical services (EMS). The study introduces EMSQA, a dataset with 24.3K questions across 10 clinical areas and 4 certification levels, along with knowledge bases containing 40K documents and 2M tokens. It also presents Expert-CoT and ExpertRAG, strategies that enhance performance by integrating clinical context, resulting in improved accuracy and exam pass rates for EMS certification.
Read full article
via arXiv — cs.CL
Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
arXiv — cs.CL2 days ago
Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
PositiveArtificial Intelligence
The article introduces the PReference Orchestrator (PRO), a framework designed to enhance the alignment of Large Language Models (LLMs) with diverse human preferences across multiple objectives. Traditional methods rely on manually set preference weights, which can hinder training efficiency and complicate user experience. PRO addresses these challenges by utilizing a lightweight preference adapter that automatically infers prompt-specific preference weights during both training and deployment, thereby improving performance and efficiency.
Read full article
via arXiv — cs.CL