Ground Truth Generation for Multilingual Historical NLP using LLMs

The introduction of STAGE (Screenplay Text, Agents, Graphs and Evaluation) marks a significant advancement in the field of narrative understanding, providing a comprehensive benchmark for evaluating knowledge graph construction, scene-level event summarization, long-context screenplay question answering, and in-script character role-playing across 150 films in English and Chinese.

A new approach called MHEL-LLaMo has been introduced for multilingual historical entity linking, utilizing a combination of a Small Language Model (SLM) and a Large Language Model (LLM). This unsupervised ensemble method addresses challenges in processing historical texts, such as linguistic variation and noisy inputs, by leveraging a multilingual bi-encoder for candidate retrieval and an instruction-tuned LLM for predictions.

A recent study introduced OrderProbe, a deterministic benchmark designed to evaluate the structural reconstruction capabilities of large language models (LLMs) using fixed four-character expressions in Chinese, Japanese, and Korean. This benchmark aims to address the challenges of sentence-level restoration from scrambled inputs, which often lack a unique solution.

A recent study analyzed the false refusal behavior of large language models (LLMs) in the context of hate speech detoxification, revealing that these models disproportionately refuse tasks involving higher semantic toxicity and specific target groups, particularly in English datasets.