Script Gap: Evaluating LLM Triage on Indian Languages in Native vs Roman Scripts in a Real World Setting

arXiv — cs.LGFriday, December 12, 2025 at 5:00:00 AM
  • A recent study evaluated the performance of Large Language Models (LLMs) in maternal and newborn healthcare triage in India, highlighting a significant performance gap between romanized and native scripts. The research found that LLMs had F1 scores 5-12 points lower for romanized messages, potentially leading to nearly 2 million excess errors in triage. This issue underscores the importance of script accuracy in high-stakes clinical applications.
  • The findings are critical for healthcare organizations in India, as the reliance on romanized text among speakers of Indian languages can compromise the effectiveness of LLMs in clinical settings. The performance degradation indicates that while LLMs can assist in healthcare, their limitations must be addressed to ensure patient safety and accurate triage outcomes.
  • This situation reflects broader challenges in the deployment of LLMs across diverse linguistic contexts, where issues of multilingual capability and script variation can lead to significant disparities in performance. The ongoing exploration of LLMs in various healthcare applications emphasizes the need for tailored approaches that consider local language practices and the potential for misalignment in AI-generated outputs.
— via World Pulse Now AI Editorial System

Was this article worth reading? Share it

Recommended apps based on your readingExplore all apps
Continue Readings
How AI-driven Sports Tech Startup ScoutEdge is Democratising Athlete Scouting in India
PositiveArtificial Intelligence
ScoutEdge, an AI-driven sports tech startup, is transforming athlete scouting in India by utilizing advanced technology to identify and promote underrated talent, thereby democratizing access to opportunities in sports. This innovation aims to bridge the gap between aspiring athletes and professional platforms.
Meta Appoints Aman Jain to Oversee Public Policy in India
PositiveArtificial Intelligence
Meta has appointed Aman Jain to lead its public policy efforts in India, bringing over two decades of experience in public policy and business strategy to the role. This strategic move is aimed at enhancing Meta's engagement with Indian regulatory frameworks and stakeholders.
2026 Could be India’s Year in AI, But Only the Resilient Will Survive
NeutralArtificial Intelligence
In 2026, India is poised to make significant advancements in artificial intelligence (AI), but experts warn that only resilient players in the sector will thrive amid increasing competition and evolving technologies. The landscape is shifting as Indian IT companies transition from embedded AI solutions to more modular products, enhancing adaptability and operational efficiency.
\textsc{Text2Graph}: Combining Lightweight LLMs and GNNs for Efficient Text Classification in Label-Scarce Scenarios
PositiveArtificial Intelligence
The newly introduced framework, Text2Graph, integrates lightweight large language models (LLMs) with graph neural networks (GNNs) to enhance text classification, particularly in scenarios with limited labels. This open-source Python package allows for flexible component swapping, including feature extractors and sampling strategies, and has been benchmarked across five datasets for zero-shot classification tasks.
A Greek Government Decisions Dataset for Public-Sector Analysis and Insight
PositiveArtificial Intelligence
An open, machine-readable dataset of Greek government decisions has been introduced, sourced from the national transparency platform Diavgeia, comprising 1 million decisions with high-quality raw text extracted from PDFs. This dataset is released with a reproducible extraction pipeline and includes qualitative analyses to explore boilerplate patterns and a retrieval-augmented generation (RAG) task to evaluate information access and reasoning over governmental documents.
LLMs in Interpreting Legal Documents
NeutralArtificial Intelligence
This chapter discusses the use of Large Language Models (LLMs) in the legal field, highlighting their ability to enhance traditional legal tasks such as interpreting statutes, contracts, and case law. It also addresses the challenges posed by these technologies, including algorithmic monoculture and compliance with regulations like the EU's AI Act and U.S. initiatives.
LMSpell: Neural Spell Checking for Low-Resource Languages
PositiveArtificial Intelligence
LMSpell has been introduced as a neural spell checking toolkit specifically designed for low-resource languages (LRLs), showcasing the effectiveness of large language models (LLMs) in improving spell correction. This toolkit includes an evaluation function that addresses the hallucination issues often associated with LLMs, marking a significant advancement in the field of natural language processing for underrepresented languages.
Local LLM Ensembles for Zero-shot Portuguese Named Entity Recognition
PositiveArtificial Intelligence
A novel approach to Named Entity Recognition (NER) for Portuguese has been introduced, utilizing a three-step ensemble pipeline of locally run Large Language Models (LLMs). This method demonstrates superior performance over individual models across multiple datasets, particularly in zero-shot scenarios, where minimal annotated data is available.

Ready to build your own newsroom?

Subscribe to unlock a personalised feed, podcasts, newsletters, and notifications tailored to the topics you actually care about