Margin-aware Preference Optimization for Aligning Diffusion Models without Reference

The introduction of QA-LIGN represents a significant advancement in the alignment of large language models (LLMs) by decomposing scalar rewards into interpretable evaluations based on principles such as helpfulness and honesty. This structured approach allows models to learn through a draft, critique, and revise pipeline, leading to improved safety and performance metrics, including a reduction in attack success rates by up to 68.7% while maintaining a low false refusal rate.

A recent study has introduced Proximalized Preference Optimization (DPO), a refined approach to direct alignment methods for large language models (LLMs). This method addresses the issue of likelihood underdetermination, which has been observed to suppress absolute likelihoods of responses, leading to unexpected model behaviors. The reformulated DPO loss allows for a broader range of feedback types and reveals the underlying causes of these limitations.

The introduction of Diffusion-SDPO, a safeguarded update rule for Direct Preference Optimization (DPO) in text-to-image diffusion models, addresses the challenge of aligning generated images with human preferences. This new approach mitigates the issue of increasing reconstruction errors in less-preferred outputs, ensuring that the quality of preferred images remains stable during optimization.

A recent study introduces SysDPO, a framework designed to align compound AI systems, which consist of multiple interacting components like large language models (LLMs) and foundation models. This approach addresses the challenges of aligning these systems with human preferences, particularly due to non-differentiable interactions and the complexity of translating system-level preferences to component-level preferences.