arXiv:2512.11682v1 Announce Type: cross 
Abstract: Therapeutic decision-making in clinical medicine constitutes a high-stakes domain in which AI guidance interacts with complex interactions among patient characteristics, disease processes, and pharmacological agents. Tasks such as drug recommendation, treatment planning, and adverse-effect prediction demand robust, multi-step reasoning grounded in reliable biomedical knowledge. Agentic AI methods, exemplified by TxAgent, address these challenges through iterative retrieval-augmented generation (RAG). TxAgent employs a fine-tuned Llama-3.1-8B model that dynamically generates and executes function calls to a unified biomedical tool suite (ToolUniverse), integrating FDA Drug API, OpenTargets, and Monarch resources to ensure access to current therapeutic information. In contrast to general-purpose RAG systems, medical applications impose stringent safety constraints, rendering the accuracy of both the reasoning trace and the sequence of tool invocations critical. These considerations motivate evaluation protocols treating token-level reasoning and tool-usage behaviors as explicit supervision signals. This work presents insights derived from our participation in the CURE-Bench NeurIPS 2025 Challenge, which benchmarks therapeutic-reasoning systems using metrics that assess correctness, tool utilization, and reasoning quality. We analyze how retrieval quality for function (tool) calls influences overall model performance and demonstrate performance gains achieved through improved tool-retrieval strategies. Our work was awarded the Excellence Award in Open Science. Complete information can be found at https://curebench.ai/.

سلطت مسابقة NeurIPS CURE-Bench الضوء على قدرات TxAgent، وهو نظام ذكاء اصطناعي مصمم لاتخاذ القرارات العلاجية في الطب السريري. باستخدام نموذج Llama-3.1-8B المعدل، يدمج TxAgent موارد طبية حيوية متنوعة، بما في ذلك واجهة برمجة تطبيقات أدوية FDA وOpenTargets، لتحسين توصيات الأدوية وتخطيط العلاج من خلال توليد معزز بالاسترجاع المتكرر.

La competencia NeurIPS CURE-Bench ha destacado las capacidades de TxAgent, un sistema de IA diseñado para la toma de decisiones terapéuticas en medicina clínica. Utilizando un modelo Llama-3.1-8B ajustado, TxAgent integra diversos recursos biomédicos, incluyendo la API de medicamentos de la FDA y OpenTargets, para mejorar las recomendaciones de medicamentos y la planificación del tratamiento a través de la generación aumentada por recuperación iterativa.

La compétition NeurIPS CURE-Bench a mis en avant les capacités de TxAgent, un système d'IA conçu pour la prise de décision thérapeutique en médecine clinique. Utilisant un modèle Llama-3.1-8B affiné, TxAgent intègre diverses ressources biomédicales, y compris l'API des médicaments de la FDA et OpenTargets, pour améliorer les recommandations de médicaments et la planification des traitements grâce à une génération augmentée par récupération itérative.

The NeurIPS CURE-Bench Competition has highlighted the capabilities of TxAgent, an AI system designed for therapeutic decision-making in clinical medicine. Utilizing a fine-tuned Llama-3.1-8B model, TxAgent integrates various biomedical resources, including the FDA Drug API and OpenTargets, to enhance drug recommendations and treatment planning through iterative retrieval-augmented generation.

MedAI: Evaluating TxAgent's Therapeutic Agentic Reasoning in the NeurIPS CURE-Bench Competition

arXiv:2512.11110v1 Announce Type: new 
Abstract: Large language models are widely used across domains, yet there are concerns about their factual reliability and biases. Factual knowledge probing offers a systematic means to evaluate these aspects. Most existing benchmarks focus on single-entity facts and monolingual data. We therefore present FIBER, a multilingual benchmark for evaluating factual knowledge in single- and multi-entity settings. The dataset includes sentence completion, question-answering, and object-count prediction tasks in English, Italian, and Turkish. Using FIBER, we examine whether the prompt language induces inference bias in entity selection and how large language models perform on multi-entity versus single-entity questions. The results indicate that the language of the prompt can influence the model's generated output, particularly for entities associated with the country corresponding to that language. However, this effect varies across different topics such that 31% of the topics exhibit factual inference bias score greater than 0.5. Moreover, the level of bias differs across languages such that Turkish prompts show higher bias compared to Italian in 83% of the topics, suggesting a language-dependent pattern. Our findings also show that models face greater difficulty when handling multi-entity questions than the single-entity questions. Model performance differs across both languages and model sizes. The highest mean average precision is achieved in English, while Turkish and Italian lead to noticeably lower scores. Larger models, including Llama-3.1-8B and Qwen-2.5-7B, show consistently better performance than smaller 3B-4B models.

تم تقديم FIBER، وهو معيار متعدد اللغات، لتقييم المعرفة الواقعية والتحيز في الاستدلال في نماذج اللغة الكبيرة بالإنجليزية والإيطالية والتركية. يتضمن هذا المعيار مهامًا مثل إكمال الجمل والإجابة على الأسئلة، بهدف تقييم كيفية تأثير لغة الموجه على اختيار الكيانات وأداء النموذج في سياقات الكيانات الفردية والمتعددة.

FIBER, un nuevo benchmark multilingüe, ha sido introducido para evaluar el conocimiento fáctico y el sesgo de inferencia en modelos de lenguaje grandes en inglés, italiano y turco. Este conjunto de datos incluye tareas como la completación de oraciones y la respuesta a preguntas, con el objetivo de evaluar cómo el idioma del prompt afecta la selección de entidades y el rendimiento del modelo en contextos de entidad única y múltiple.

FIBER, une nouvelle référence multilingue, a été introduite pour évaluer les connaissances factuelles et les biais d'inférence dans les grands modèles de langage en anglais, en italien et en turc. Ce jeu de données comprend des tâches telles que la complétion de phrases et le questionnement, visant à évaluer comment la langue du prompt influence la sélection des entités et la performance des modèles dans des contextes à entité unique et multiple.

FIBER, a new multilingual benchmark, has been introduced to evaluate factual knowledge and inference bias in large language models across English, Italian, and Turkish. This dataset includes tasks such as sentence completion and question-answering, aiming to assess how prompt language affects entity selection and model performance in single- and multi-entity contexts.

FIBER: A Multilingual Evaluation Resource for Factual Inference Bias

One More Thing in AI – Your Shortcut to AI Mastery

MedAI: Evaluating TxAgent's Therapeutic Agentic Reasoning in the NeurIPS CURE-Bench Competition

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

Supanote

AI Agent Catalog

DBTAI

Aidx.ai

Ready to build your own newsroom?