GRUFF: LLM Pronoun Fidelity, Reasoning, and Biases in German

A new study introduces GRUFF, a large-scale dataset designed to evaluate pronoun fidelity in German, focusing on the use of third-person singular pronouns and their grammatical agreement across various gender systems. This research highlights the capabilities of large language models (LLMs) in handling gendered language, particularly in German, which has more complex gender agreements compared to English.

The findings are significant as they reveal that LLMs demonstrate strong grammatical agreement for masculine and feminine entities when explicit context is absent, indicating their potential for improved language processing in gendered contexts.

This development aligns with ongoing discussions about the performance of LLMs in multilingual settings, particularly their ability to adapt to cultural nuances and reasoning tasks across different languages, which remains a critical area of research in artificial intelligence.