Periodic Asynchrony: An Effective Method for Accelerating On-Policy Reinforcement Learning

Recent research has focused on improving the honesty of language models in deductive reasoning, emphasizing their ability to provide answers only when logically entailed by the premises. The study introduces multi-step tasks and datasets to evaluate this capability, revealing that existing training methods struggle with these challenges.

ExPO-HM (Explain-then-Detect Policy Optimization for Hateful Memes) has been proposed to enhance the detection of hateful memes, addressing limitations in existing models that primarily provide binary predictions without context. This new approach aims to incorporate reasoning similar to human annotators, improving the understanding of policy-relevant cues such as targets and attack types.

A new framework called ReVeL (Rewrite and Verify by LLM) has been proposed to enhance the multiple-choice question answering (MCQA) format used in evaluating multimodal language models. This framework transforms MCQA into open-form questions while ensuring answers remain verifiable, addressing issues of answer guessing and unreliable accuracy metrics during reinforcement fine-tuning (RFT).

The introduction of VADE, a Variance-Aware Dynamic Sampling framework, aims to enhance group-based policy optimization methods in multimodal reinforcement learning (RL) by addressing the gradient vanishing problem. This issue arises when identical rewards are assigned to all responses within a group, leading to diminished training signals. VADE proposes an online sample-level difficulty estimation to improve the selection of effective samples during training.

The SPINE framework introduces a token-selective approach to test-time reinforcement learning, addressing the challenges faced by large language models (LLMs) and multimodal LLMs (MLLMs) during distribution shifts at test-time. By focusing on high-entropy tokens and applying an entropy-band regularizer, SPINE aims to enhance model performance and maintain exploration during reinforcement learning processes.

The introduction of Perceptual-Evidence Anchored Reinforced Learning (PEARL) marks a significant advancement in multimodal reasoning, addressing the limitations of traditional Reinforcement Learning with Verifiable Rewards (RLVR) in Vision-Language Models (VLMs). PEARL enhances reasoning by anchoring it to verified visual evidence, thus mitigating issues like visual hallucinations and reward hacking.

The introduction of Bayesian Prior-Guided Optimization (BPGO) enhances Group Relative Policy Optimization (GRPO) by addressing the inherent ambiguity in visual generation tasks. BPGO incorporates a semantic prior anchor to model reward uncertainty, allowing for more effective optimization by emphasizing reliable feedback while down-weighting ambiguous signals.

A new method called Syn-GRPO (Synthesis-GRPO) has been proposed to enhance the reinforcement learning capabilities of Multimodal Large Language Models (MLLMs) by synthesizing high-quality training data through an online data generator. This approach aims to address the existing challenges of low data quality that limit the exploration scope in MLLM training.