Audio Question Answering with GRPO-Based Fine-Tuning and Calibrated Segment-Level Predictions

Membership inference attacks (MIAs) on large language models (LLMs) present significant privacy risks during model training. Recent advancements in Reinforcement Learning with Verifiable Rewards (RLVR) have transformed LLM training, especially for complex reasoning tasks. However, the on-policy nature of RLVR leads to unique privacy concerns, as it requires determining if a prompt was used in fine-tuning, creating potential leakage not from memorization but from behavioral changes. The Divergence-in-Behavior Attack (DIBA) framework is proposed to address this risk.

As AI systems advance, deceptive behaviors pose challenges in evaluation and user trust. Recent research indicates that lie detectors can effectively identify deception, yet they are seldom integrated into training due to fears of contamination and manipulation. This study explores the impact of incorporating lie detectors in the labeling phase of large language model (LLM) training, using a new dataset called DolusChat. It identifies key factors influencing the honesty of learned policies, revealing that preference learning with lie detectors can lead to evasion strategies.