QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA

The introduction of Semantic Soft Bootstrapping (SSB) represents a significant advancement in long context reasoning for large language models (LLMs), allowing them to enhance cognitive capabilities without relying on reinforcement learning with verifiable rewards (RLVR). This self-distillation technique enables the model to act as both teacher and student, improving its reasoning abilities through varied semantic contexts during training.

The recent study on Group Relative Policy Optimization (GRPO) in Search-R1 highlights a significant issue known as Lazy Likelihood Displacement (LLD), which leads to a collapse in training effectiveness. This phenomenon results in a self-reinforcing cycle of declining response quality, characterized by low-confidence outputs and inflated gradients. The research empirically demonstrates this collapse across various models engaged in search-integrated question answering tasks.

A new paradigm for Image Quality Assessment (IQA) has been introduced, focusing on the aesthetic quality of interior images through a framework called Spatial Aesthetics. This framework evaluates images based on layout, harmony, lighting, and distortion, supported by the SA-BENCH benchmark, which includes 18,000 images and 50,000 annotations. The SA-IQA methodology has been developed to enhance the assessment of AI-generated images (AIGI) and is applied in optimizing generation pipelines and selecting high-quality outputs.

A recent study has introduced Proximalized Preference Optimization (DPO), a refined approach to direct alignment methods for large language models (LLMs). This method addresses the issue of likelihood underdetermination, which has been observed to suppress absolute likelihoods of responses, leading to unexpected model behaviors. The reformulated DPO loss allows for a broader range of feedback types and reveals the underlying causes of these limitations.

A new approach called margin-aware preference optimization (MaPO) has been introduced to address the challenges of reference mismatch in aligning text-to-image diffusion models. This method allows for effective adaptation without relying on a reference model, which has been a limitation in existing preference alignment techniques like Direct Preference Optimization (DPO).

A recent study investigates the impact of explicit world-modeling objectives on the internal representations and performance of Transformers, particularly in the context of a controlled Rubik's Cube task. The research compares standard next-token prediction with two world-modeling strategies, revealing that explicit modeling enhances representation quality and downstream performance after reinforcement learning post-training.

DVPO, or Distributional Value Modeling-based Policy Optimization, has been introduced as a new reinforcement learning framework aimed at enhancing the post-training phase of large language models (LLMs). This framework addresses the challenges posed by noisy supervision and aims to improve both robustness and generalization by utilizing conditional risk theory and token-level value distributions.

AdaptVision has been introduced as a new paradigm in Vision-Language Models (VLMs), focusing on adaptive visual token acquisition to enhance efficiency in visual question answering tasks. By employing a coarse-to-fine approach, the model selectively acquires visual information as needed, addressing the computational overhead associated with traditional methods that rely on fixed-ratio compression.