World PulseNow
Newsletter

Large Language Model Benchmarks in Medical Tasks

arXiv — cs.CL•Thursday, November 13, 2025 at 5:00:00 AM
PositiveArtificial Intelligence
The increasing application of large language models (LLMs) in the medical domain necessitates robust evaluation methods, particularly through benchmark datasets. A recent survey presents a comprehensive overview of various datasets utilized in medical LLM tasks, categorizing them by modality and discussing their significance in clinical applications. Key benchmarks such as MIMIC-III, MIMIC-IV, BioASQ, PubMedQA, and CheXpert have been pivotal in advancing tasks like medical report generation and clinical summarization. These datasets not only facilitate the development of LLMs but also highlight the challenges faced in achieving greater language diversity and innovative data synthesis methods. The findings underscore the importance of these benchmarks in shaping future research directions, ultimately enhancing the capabilities of multimodal medical intelligence.
— via World Pulse Now AI Editorial System
Read Original

Was this article worth reading? Share it

Recommended Readings
Silenced Biases: The Dark Side LLMs Learned to Refuse
arXiv — stat.ML19 hours ago
Silenced Biases: The Dark Side LLMs Learned to Refuse
NegativeArtificial Intelligence
Safety-aligned large language models (LLMs) are increasingly used in sensitive applications where fairness is crucial. Evaluating their fairness is complex, often relying on standard question-answer schemes that may misinterpret refusal responses as indicators of fairness. This paper introduces the concept of silenced biases, which are unfair preferences hidden within the models' latent space, masked by safety-alignment. Previous methods have limitations, prompting the need for a new approach to assess these biases effectively.
Read full article
via arXiv — stat.ML
Fair In-Context Learning via Latent Concept Variables
arXiv — cs.LG19 hours ago
Fair In-Context Learning via Latent Concept Variables
PositiveArtificial Intelligence
The paper titled 'Fair In-Context Learning via Latent Concept Variables' explores the in-context learning (ICL) capabilities of large language models (LLMs) and their potential biases when applied to tabular data. It emphasizes an optimal demonstration selection method that leverages latent concept variables to enhance task adaptation while promoting fairness. The study introduces data augmentation strategies aimed at minimizing correlations between sensitive variables and predictive outcomes, ultimately striving for equitable predictions.
Read full article
via arXiv — cs.LG
Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models
arXiv — cs.LG19 hours ago
Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models
PositiveArtificial Intelligence
The paper discusses the challenges of test-time prompt tuning for vision-language models, highlighting the issue of prompt optimization bias that can lead to suboptimal performance. It identifies two main causes: the model's focus on entropy minimization, which may overlook prediction accuracy, and data misalignment between visual and textual modalities. To address these issues, the authors propose a new method called Doubly Debiased Test-Time Prompt Tuning, aimed at improving model performance in downstream tasks.
Read full article
via arXiv — cs.LG
Modeling and Predicting Multi-Turn Answer Instability in Large Language Models
arXiv — cs.CL2 days ago
Modeling and Predicting Multi-Turn Answer Instability in Large Language Models
NeutralArtificial Intelligence
The paper titled 'Modeling and Predicting Multi-Turn Answer Instability in Large Language Models' discusses the evaluation of large language models (LLMs) in terms of their robustness during user interactions. The study employs multi-turn follow-up prompts to assess changes in model answers and accuracy dynamics using Markov chains. Results indicate vulnerabilities in LLMs, with a 10% accuracy drop for Gemini 1.5 Flash after a 'Think again' prompt over nine turns, and a 7.5% drop for Claude 3.5 Haiku with a reworded question. The findings suggest that accuracy can be modeled over time.
Read full article
via arXiv — cs.CL
Who Gets the Reward, Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents
arXiv — cs.CL2 days ago
Who Gets the Reward, Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents
PositiveArtificial Intelligence
The article discusses a new theoretical framework for training multi-agent systems using large language models (LLMs). It aims to connect system-level evaluations with agent-level learning by integrating cooperative game-theoretic attribution and process reward modeling. This approach produces local, signed, and credit-conserving signals, enhancing cooperation among agents while penalizing harmful actions in failure scenarios.
Read full article
via arXiv — cs.CL
Identifying and Analyzing Performance-Critical Tokens in Large Language Models
arXiv — cs.CL2 days ago
Identifying and Analyzing Performance-Critical Tokens in Large Language Models
NeutralArtificial Intelligence
The paper titled 'Identifying and Analyzing Performance-Critical Tokens in Large Language Models' explores how large language models (LLMs) utilize in-context learning (ICL) for few-shot learning. It categorizes tokens in ICL prompts into content, stopword, and template tokens, aiming to identify those that significantly impact LLM performance. The study reveals that template and stopword tokens have a greater influence on performance than informative content tokens, challenging existing assumptions about human attention to informative words.
Read full article
via arXiv — cs.CL
A Multifaceted Analysis of Negative Bias in Large Language Models through the Lens of Parametric Knowledge
arXiv — cs.CL2 days ago
A Multifaceted Analysis of Negative Bias in Large Language Models through the Lens of Parametric Knowledge
NeutralArtificial Intelligence
A recent study published on arXiv examines the phenomenon of negative bias in large language models (LLMs), which refers to their tendency to generate negative responses in binary decision tasks. The research highlights that previous studies have primarily focused on identifying negative attention heads that contribute to this bias. The authors introduce a new evaluation pipeline that categorizes responses based on the model's parametric knowledge, revealing that the format of prompts significantly influences the responses more than the semantics of the content itself.
Read full article
via arXiv — cs.CL
LDC: Learning to Generate Research Idea with Dynamic Control
arXiv — cs.CL2 days ago
LDC: Learning to Generate Research Idea with Dynamic Control
PositiveArtificial Intelligence
Recent advancements in large language models (LLMs) highlight their potential in automating scientific research ideation. Current methods often produce ideas that do not meet expert standards of novelty, feasibility, and effectiveness. To address these issues, a new framework is proposed that combines Supervised Fine-Tuning (SFT) and controllable Reinforcement Learning (RL) to enhance the quality of generated research ideas through a two-stage approach.
Read full article
via arXiv — cs.CL