Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap

The article introduces DeepEL, a new framework for Entity Linking (EL) that integrates Large Language Models (LLMs) at every stage of the EL process. This approach aims to enhance natural language understanding by improving entity disambiguation and input representation. Previous methods often applied LLMs in isolation, limiting their effectiveness. DeepEL addresses this by proposing a self-validation mechanism that leverages global context, thus aiming for greater accuracy and robustness in entity linking tasks.

This study analyzes the transformative role of Large Language Models (LLMs) in research and development (R&D) processes. By automating knowledge discovery, enhancing hypothesis generation, and fostering collaboration within innovation ecosystems, LLMs significantly improve research efficiency and effectiveness. The research highlights how LLMs facilitate more adaptable and informed R&D workflows, ultimately accelerating innovation cycles and reducing time-to-market for groundbreaking ideas.

The article presents a novel training strategy called Curriculum-based Relative Policy Optimization (CuRPO) aimed at improving Visual Grounding tasks. It highlights the limitations of Chain-of-Thought (CoT) prompting, particularly when outputs become lengthy or complex, which can degrade performance. The study reveals that simply increasing dataset size does not guarantee better results due to varying complexities. CuRPO utilizes CoT length and generalized Intersection over Union (gIoU) rewards to structure training data progressively from simpler to more challenging examples, demonstrating ef…

Self-Examining Reinforcement Learning (SERL) is a proposed framework that addresses challenges in applying Reinforcement Learning (RL) to open-domain tasks. Traditional methods face issues with subjectivity and reliance on external rewards. SERL innovatively positions large language models (LLMs) as both Actor and Judge, utilizing internal reward mechanisms. It employs Copeland-style pairwise comparisons to enhance the Actor's capabilities and introduces a self-consistency reward to improve the Judge's reliability, aiming to advance RL applications in open domains.

MoHoBench is a newly developed benchmark aimed at assessing the honesty of Multimodal Large Language Models (MLLMs) when confronted with unanswerable visual questions. Despite advancements in vision-language tasks, MLLMs often produce unreliable content. This study systematically evaluates the honesty of 28 popular MLLMs using a dataset of over 12,000 visual questions, revealing that many models struggle to provide honest responses. The findings highlight the need for improved trustworthiness in AI systems.

MedBench v4 is a new benchmarking infrastructure designed to evaluate Chinese medical language models, multimodal models, and intelligent agents. It features over 700,000 expert-curated tasks across various specialties, with evaluations conducted by clinicians from more than 500 institutions. The study assessed 15 advanced models, revealing that base LLMs scored an average of 54.1/100, while safety and ethics ratings were notably low at 18.4/100. Multimodal models performed even worse, indicating a need for improved evaluation frameworks in medical AI.

The integration of Large Language Models (LLMs) with 3D vision is revolutionizing robotic perception and autonomy. This approach enhances robotic sensing technologies, allowing machines to understand and interact with complex environments using natural language and spatial awareness. The review discusses the foundational principles of LLMs and 3D data, examines critical 3D sensing technologies, and highlights advancements in scene understanding, text-to-3D generation, and embodied agents, while addressing the challenges faced in this evolving field.

The research paper explores the challenge of false information and the effectiveness of large language models (LLMs) in verifying factual claims in English and Telugu. It presents a bilingual dataset and evaluates various approaches for classifying the veracity of claims. The study aims to enhance the efficiency of fact-checking processes, which are often labor-intensive and time-consuming.