Preference Learning with Lie Detectors can Induce Honesty or Evasion

This report details a submission to Track 5 of the DCASE 2025 Challenge focused on Audio Question Answering (AQA). The system utilizes the SSL backbone BEATs to extract frame-level audio features, which are processed by a classification head to generate segment-level predictions of acoustic events based on the Audioset ontology. These predictions are calibrated before producing event-level predictions, which are then structured into a prompt for a fine-tuned version of Qwen2.5-7B-Instruct, trained with the GRPO algorithm. The method achieved an accuracy of 62.6% on the development set, highlig…

The article discusses a fine-tuning task for large language models (LLMs) aimed at creating reward functions for sequential resource allocation in public health, influenced by human preferences in natural language. This task presents challenges due to complex objectives and limited data. The authors introduce DPO-PRO, a fine-tuning algorithm based on Direct Preference Optimization (DPO) that utilizes a lightweight Distributionally Robust Optimization (DRO) approach to handle uncertainty in preference distribution. DPO-PRO shows improved robustness against noisy preference signals in evaluation…

This study evaluates the understanding of road safety principles by multi-modal large language models (LLMs), particularly in the context of autonomous vehicles. Using a curated dataset of traffic signs and safety norms from school textbooks, the research reveals that these models struggle with safety reasoning, highlighting significant gaps between human learning and model interpretation. The findings suggest a need for further research to address these performance deficiencies in AI systems governing autonomous vehicles.

Membership inference attacks (MIAs) on large language models (LLMs) present significant privacy risks during model training. Recent advancements in Reinforcement Learning with Verifiable Rewards (RLVR) have transformed LLM training, especially for complex reasoning tasks. However, the on-policy nature of RLVR leads to unique privacy concerns, as it requires determining if a prompt was used in fine-tuning, creating potential leakage not from memorization but from behavioral changes. The Divergence-in-Behavior Attack (DIBA) framework is proposed to address this risk.