Understanding Safety-Sensitive Expert Behavior in Mixture-of-Experts LLMs

Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior

NeutralArtificial Intelligence

Recent research has highlighted the importance of psychometric evaluation in large language models (LLMs), particularly focusing on the reliability of self-reports in predicting behavior. The study contrasts traditional personality assessments, like the Big 5, with the Theory of Planned Behavior (TPB), demonstrating that self-report coherence exists but is context-dependent.

Operadic consistency: a label-free signal for compositional reasoning failures in LLMs

NeutralArtificial Intelligence

A recent study introduced the concept of operadic consistency (OC) as a method to detect reasoning failures in large language models (LLMs) during inference without relying on ground-truth labels. This approach correlates strongly with accuracy across multiple multi-hop question-answering datasets, suggesting that a model's direct answer should align with its compositional reasoning outputs.

When Similar Means Different: Evaluating LLMs on Arabic--Hebrew Cognates

NeutralArtificial Intelligence

A new benchmark called SemCog Bench has been introduced to evaluate large language models (LLMs) on Arabic-Hebrew cognates, consisting of 1,858 word pairs with annotations for cognate identification and semantic disambiguation. The study reveals a significant performance gap in cross-lingual reasoning, particularly with false friends and loanwords, where models struggle despite high accuracy on true cognates.

Evaluating Pluralism in LLMs through Latent Perspectives

NeutralArtificial Intelligence

A recent study published on arXiv introduces a multi-layered framework for the unsupervised extraction of perspectives in large language models (LLMs), aiming to address the challenges of pluralistic alignment in LLM-generated text. The framework was evaluated using book reviews, a dataset rich in diverse opinions, to identify the pluralistic gap in LLM outputs.

WildIFEval: Instruction Following in the Wild

NeutralArtificial Intelligence

A new dataset named WildIFEval has been introduced, comprising 7,000 real user instructions characterized by diverse multi-constraint conditions, aimed at enhancing the instruction-following capabilities of large language models (LLMs). This dataset categorizes constraints into eight high-level classes, providing a comprehensive framework for benchmarking LLM performance.

One Polluted Page Is Enough: Evaluating Web Content Pollution in Generative Recommenders

NegativeArtificial Intelligence

A recent study highlights the risks associated with search-augmented large language models (LLMs) that may inadvertently promote fake products due to polluted web content, such as misleading reviews and promotional pages. The research introduces FORGE, a benchmark designed to evaluate the extent of fake product promotion by these generative recommenders.

Constrained Semantic Decompression in LLMs through Persian Proverb-Conditioned Story Generation

NeutralArtificial Intelligence

Recent research has introduced a novel approach to transforming Persian proverbs into engaging narratives through a method termed constrained semantic decompression. This study utilizes the Proverb Aligned Narrative Dataset (PAND), which pairs proverbs with human-written stories, highlighting the challenges faced by large language models (LLMs) in accurately capturing the moral and causal structures embedded in these proverbs.

Influcoder: Distilling Decoders' Gradient Influence Rankings into an Encoder for Data Attribution

NeutralArtificial Intelligence

A new method called Influcoder has been proposed to enhance Data Attribution (DA) in large language models (LLMs) by efficiently estimating the influence of individual training samples on model outputs. This approach addresses the limitations of existing influence function methods, which struggle with speed and storage when applied to large datasets.