arXiv:2511.10658v1 Announce Type: new 
Abstract: Large language models (LLMs) are increasingly used to extract structured information from free-text clinical records, but prior work often focuses on single tasks, limited models, and English-language reports. We evaluated 15 open-weight LLMs on pathology and radiology reports across six use cases, colorectal liver metastases, liver tumours, neurodegenerative diseases, soft-tissue tumours, melanomas, and sarcomas, at three institutes in the Netherlands, UK, and Czech Republic. Models included general-purpose and medical-specialised LLMs of various sizes, and six prompting strategies were compared: zero-shot, one-shot, few-shot, chain-of-thought, self-consistency, and prompt graph. Performance was assessed using task-appropriate metrics, with consensus rank aggregation and linear mixed-effects models quantifying variance. Top-ranked models achieved macro-average scores close to inter-rater agreement across tasks. Small-to-medium general-purpose models performed comparably to large models, while tiny and specialised models performed worse. Prompt graph and few-shot prompting improved performance by ~13%. Task-specific factors, including variable complexity and annotation variability, influenced results more than model size or prompting strategy. These findings show that open-weight LLMs can extract structured data from clinical reports across diseases, languages, and institutions, offering a scalable approach for clinical data curation.

تستخدم نماذج اللغة الكبيرة (LLMs) بشكل متزايد لاستخراج المعلومات المنظمة من السجلات السريرية. تم تقييم 15 نموذجًا مفتوح الوزن مؤخرًا، مع التركيز على تقارير علم الأمراض والأشعة عبر ست حالات استخدام، بما في ذلك النقائل الكبدية القولونية والأمراض التنكسية العصبية، في هولندا والمملكة المتحدة وجمهورية التشيك. قارن البحث استراتيجيات التحفيز المختلفة ووجد أن النماذج الأعلى حققت درجات متوسطة قريبة من اتفاقية المقيمين، مما يدل على فعاليتها في استخراج البيانات المنظمة.

Los grandes modelos de lenguaje (LLMs) se utilizan cada vez más para extraer información estructurada de los registros clínicos. Una evaluación reciente de 15 LLMs de peso abierto se centró en informes de patología y radiología en seis casos de uso, incluidos metástasis hepáticas colorrectales y enfermedades neurodegenerativas, realizada en los Países Bajos, el Reino Unido y la República Checa. El estudio comparó varias estrategias de solicitud y encontró que los mejores modelos lograron puntajes promedio cercanos al acuerdo entre evaluadores, lo que indica su efectividad en la extracción de d…

Les grands modèles de langage (LLMs) sont de plus en plus utilisés pour extraire des informations structurées des dossiers cliniques. Une récente évaluation de 15 LLMs à poids ouvert s'est concentrée sur des rapports de pathologie et de radiologie à travers six cas d'utilisation, y compris les métastases hépatiques colorectales et les maladies neurodégénératives, réalisée aux Pays-Bas, au Royaume-Uni et en République tchèque. L'étude a comparé diverses stratégies de sollicitation et a constaté que les meilleurs modèles atteignaient des scores moyens proches de l'accord inter-évaluateurs, indiq…

Large language models (LLMs) are increasingly utilized for extracting structured information from clinical records. A recent evaluation of 15 open-weight LLMs focused on pathology and radiology reports across six use cases, including colorectal liver metastases and neurodegenerative diseases, conducted in the Netherlands, UK, and Czech Republic. The study compared various prompting strategies and found that top models achieved macro-average scores close to inter-rater agreement, indicating their effectiveness in structured data extraction.

Evaluating Open-Weight Large Language Models for Structured Data Extraction from Narrative Medical Reports Across Multiple Use Cases and Languages

One More Thing in AI – Your Shortcut to AI Mastery

Evaluating Open-Weight Large Language Models for Structured Data Extraction from Narrative Medical Reports Across Multiple Use Cases and Languages

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

Supametas.AI

Resub

OpenL Translator

DocsLoop

Ready to build your own newsroom?