arXiv:2506.00299v2 Announce Type: replace 
Abstract: Diffusion models are state-of-the-art generative models, yet their samples often fail to satisfy application objectives such as safety constraints or domain-specific validity. Existing techniques for alignment require gradients, internal model access, or large computational budgets resulting in high compute demands, or lack of support for certain objectives. In response, we introduce an inference-time alignment framework based on evolutionary algorithms. We treat diffusion models as black boxes and search their latent space to maximize alignment objectives. Given equal or less running time, our method achieves 3-35% higher ImageReward scores than gradient-free and gradient-based methods. On the Open Image Preferences dataset, our method achieves competitive results across four popular alignment objectives. In terms of computational efficiency, we require 55% to 76% less GPU memory and are 72% to 80% faster than gradient-based methods.

تم تقديم إطار عمل جديد لمحاذاة نماذج الانتشار أثناء الاستدلال، باستخدام خوارزميات تطورية لتحسين الأداء دون الحاجة إلى موارد حسابية كبيرة. تعالج هذه الطريقة نماذج الانتشار كصناديق سوداء وتقوم بتحسين فضاءها الكامن لتحقيق أهداف محاذاة محددة، محققة درجات أعلى من ImageReward مقارنة بالتقنيات الحالية.

Se ha introducido un nuevo marco para alinear modelos de difusión durante la inferencia, utilizando algoritmos evolutivos para mejorar el rendimiento sin requerir recursos computacionales extensos. Este método trata a los modelos de difusión como cajas negras y optimiza su espacio latente para cumplir con objetivos de alineación específicos, logrando puntajes de ImageReward significativamente más altos en comparación con las técnicas existentes.

Un nouveau cadre pour l'alignement des modèles de diffusion lors de l'inférence a été introduit, utilisant des algorithmes évolutionnaires pour améliorer les performances sans nécessiter de ressources computationnelles importantes. Cette méthode traite les modèles de diffusion comme des boîtes noires et optimise leur espace latent pour répondre à des objectifs d'alignement spécifiques, atteignant des scores ImageReward significativement plus élevés par rapport aux techniques existantes.

A new framework for aligning diffusion models during inference has been introduced, utilizing evolutionary algorithms to enhance performance without requiring extensive computational resources. This method treats diffusion models as black boxes and optimizes their latent space to meet specific alignment objectives, achieving significantly higher ImageReward scores compared to existing techniques.

Inference-Time Alignment of Diffusion Models via Evolutionary Algorithms

arXiv:2512.02019v1 Announce Type: cross 
Abstract: Diffusion models have achieved remarkable success in data-driven learning and in sampling from complex, unnormalized target distributions. Building on this progress, we reinterpret Maximum Entropy Reinforcement Learning (MaxEntRL) as a diffusion model-based sampling problem. We tackle this problem by minimizing the reverse Kullback-Leibler (KL) divergence between the diffusion policy and the optimal policy distribution using a tractable upper bound. By applying the policy gradient theorem to this objective, we derive a modified surrogate objective for MaxEntRL that incorporates diffusion dynamics in a principled way. This leads to simple diffusion-based variants of Soft Actor-Critic (SAC), Proximal Policy Optimization (PPO) and Wasserstein Policy Optimization (WPO), termed DiffSAC, DiffPPO and DiffWPO. All of these methods require only minor implementation changes to their base algorithm. We find that on standard continuous control benchmarks, DiffSAC, DiffPPO and DiffWPO achieve better returns and higher sample efficiency than SAC and PPO.

تم تقديم إطار عمل جديد يعيد تفسير التعلم المعزز بأقصى انتروبيا (MaxEntRL) كمشكلة أخذ عينات قائمة على نماذج الانتشار، بهدف تقليل تباين كولباك-ليبلر العكسي بين سياسة الانتشار وتوزيع السياسة المثلى. تقود هذه الطريقة إلى تطوير متغيرات قائمة على الانتشار لخوارزميات موجودة مثل Soft Actor-Critic (SAC) وProximal Policy Optimization (PPO) وWasserstein Policy Optimization (WPO).

Se ha introducido un nuevo marco que reinterpreta el Aprendizaje por Refuerzo de Entropía Máxima (MaxEntRL) como un problema de muestreo basado en modelos de difusión, con el objetivo de minimizar la divergencia Kullback-Leibler inversa entre la política de difusión y la distribución de política óptima. Este enfoque conduce al desarrollo de variantes basadas en difusión de algoritmos existentes como Soft Actor-Critic (SAC), Proximal Policy Optimization (PPO) y Wasserstein Policy Optimization (WPO).

Un nouveau cadre a été introduit, réinterprétant l'apprentissage par renforcement à entropie maximale (MaxEntRL) comme un problème d'échantillonnage basé sur des modèles de diffusion, visant à minimiser la divergence Kullback-Leibler inverse entre la politique de diffusion et la distribution de politique optimale. Cette approche conduit au développement de variantes basées sur la diffusion d'algorithmes existants tels que Soft Actor-Critic (SAC), Proximal Policy Optimization (PPO) et Wasserstein Policy Optimization (WPO).

A new framework has been introduced that reinterprets Maximum Entropy Reinforcement Learning (MaxEntRL) as a diffusion model-based sampling problem, aiming to minimize the reverse Kullback-Leibler divergence between the diffusion policy and the optimal policy distribution. This approach leads to the development of diffusion-based variants of existing algorithms such as Soft Actor-Critic (SAC), Proximal Policy Optimization (PPO), and Wasserstein Policy Optimization (WPO).

Inference-Time Alignment of Diffusion Models via Evolutionary Algorithms

Was this article worth reading? Share it

AINF Membership

Deptho.ai

Blunge