arXiv:2507.21802v3 Announce Type: replace-cross 
Abstract: Although GRPO substantially enhances flow matching models in human preference alignment of image generation, methods such as FlowGRPO and DanceGRPO still exhibit inefficiency due to the necessity of sampling and optimizing over all denoising steps specified by the Markov Decision Process (MDP). In this paper, we propose $\textbf{MixGRPO}$, a novel framework that leverages the flexibility of mixed sampling strategies through the integration of stochastic differential equations (SDE) and ordinary differential equations (ODE). This streamlines the optimization process within the MDP to improve efficiency and boost performance. Specifically, MixGRPO introduces a sliding window mechanism, using SDE sampling and GRPO-guided optimization only within the window, while applying ODE sampling outside. This design confines sampling randomness to the time-steps within the window, thereby reducing the optimization overhead, and allowing for more focused gradient updates to accelerate convergence. Additionally, as time-steps beyond the sliding window are not involved in optimization, higher-order solvers are supported for faster sampling. So we present a faster variant, termed $\textbf{MixGRPO-Flash}$, which further improves training efficiency while achieving comparable performance. MixGRPO exhibits substantial gains across multiple dimensions of human preference alignment, outperforming DanceGRPO in both effectiveness and efficiency, with nearly 50% lower training time. Notably, MixGRPO-Flash further reduces training time by 71%.

يمثل تقديم MixGRPO تقدمًا كبيرًا في نماذج التوليد المعتمدة على التدفق، حيث يعزز كفاءة توافق تفضيلات البشر في توليد الصور من خلال دمج استراتيجيات أخذ العينات المختلطة مع المعادلات التفاضلية العشوائية والعادية. يهدف هذا الإطار الجديد إلى تبسيط عملية التحسين ضمن عملية اتخاذ القرار ماركوف، مما يقلل من الأعباء الحاسوبية مع تحسين الأداء.

La introducción de MixGRPO marca un avance significativo en los modelos generativos basados en flujo, mejorando la eficiencia de la alineación de preferencias humanas en la generación de imágenes al integrar estrategias de muestreo mixtas con ecuaciones diferenciales estocásticas y ordinarias. Este nuevo marco busca optimizar el proceso dentro del Proceso de Decisión de Markov, reduciendo la sobrecarga computacional mientras mejora el rendimiento.

L'introduction de MixGRPO représente une avancée significative dans les modèles génératifs basés sur le flux, améliorant l'efficacité de l'alignement des préférences humaines dans la génération d'images en intégrant des stratégies d'échantillonnage mixtes avec des équations différentielles stochastiques et ordinaires. Ce nouveau cadre vise à rationaliser le processus d'optimisation au sein du processus de décision de Markov, réduisant les coûts computationnels tout en améliorant les performances.

The introduction of MixGRPO marks a significant advancement in flow-based generative models, enhancing the efficiency of human preference alignment in image generation by integrating mixed sampling strategies with stochastic and ordinary differential equations. This novel framework aims to streamline the optimization process within the Markov Decision Process, reducing computational overhead while improving performance.

MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE

Was this article worth reading? Share it

LucidQuery AI

SwapAnything.io

RepoFlow

Metaflow AI

Gonogo by Trampoline

MixflowAI

Ready to build your own newsroom?