arXiv:2603.17685v3 Announce Type: replace 
Abstract: Balancing policy expressiveness with the exploration-exploitation trade-off is a core challenge in online Reinforcement Learning (RL). While Stochastic Differential Equation (SDE)-based diffusion policies can represent complex, multimodal action distributions, they suffer from two critical limitations: their stochastic reverse processes render entropy intractable (necessitating heuristic exploration), and computing policy gradients through long denoising chains is expensive and unstable. In this work, we show that ODE-based flow matching inherently resolves these issues by enabling both simulation-free policy optimization and tractable entropy computation. Building on this, we introduce Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (FMER). Our framework exploits this insight in three ways. First, we theoretically establish that minimizing an advantage-weighted conditional flow matching loss acts as a simulation-free surrogate for policy mirror descent. This steers the velocity field toward high-value regions while entirely avoiding backpropagation through the ODE solver. Second, we derive an analytic entropy objective that corrects for the density distortion caused by the $\tanh$ transformation (mapping an unbounded latent space to bounded actions), thereby facilitating principled maximum-entropy optimization. Finally, we dynamically tune the mirror descent temperature based on the effective sample size to enforce a robust trust region during training. Empirical evaluations demonstrate that FMER achieves superior performance on the challenging sparse-reward FrankaKitchen environment, while maintaining competitive results across standard dense-reward MuJoCo benchmarks.

قدمت دراسة حديثة تحسين سياسة مطابقة التدفق باستخدام الانحدار المرآتي وقيود الانتروبيا (FMER)، حيث تعالج التحديات الرئيسية في التعلم المعزز (RL) من خلال تمكين تحسين السياسة بدون محاكاة وحساب الانتروبيا القابل للإدارة. يتجاوز هذا الإطار القيود المفروضة على السياسات القائمة على المعادلات التفاضلية العشوائية، التي تواجه صعوبات مع عدم إمكانية حساب الانتروبيا وتكاليف تدرجات السياسة.

Un estudio reciente ha presentado la Optimización de Políticas por Correspondencia de Flujo con Descenso Espejo y Restricciones de Entropía (FMER), abordando desafíos clave en el Aprendizaje por Refuerzo (RL) al permitir la optimización de políticas sin simulación y el cálculo de entropía manejable. Este marco supera las limitaciones de las políticas de difusión basadas en Ecuaciones Diferenciales Estocásticas, que enfrentan dificultades con la intractabilidad de la entropía y los costosos gradientes de políticas.

Une étude récente a introduit l'optimisation de politique par correspondance de flux avec descente miroir et contraintes d'entropie (FMER), abordant des défis clés dans l'apprentissage par renforcement (RL) en permettant une optimisation de politique sans simulation et un calcul d'entropie tractable. Ce cadre surmonte les limitations des politiques de diffusion basées sur des équations différentielles stochastiques, qui rencontrent des difficultés avec l'intractabilité de l'entropie et des gradients de politique coûteux.

A recent study has introduced Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (FMER), addressing key challenges in online Reinforcement Learning (RL) by enabling simulation-free policy optimization and tractable entropy computation. This framework overcomes limitations of Stochastic Differential Equation-based diffusion policies, which struggle with entropy intractability and expensive policy gradients.