arXiv:2602.02192v5 Announce Type: replace 
Abstract: Reinforcement learning (RL) is a critical stage in post-training large language models (LLMs), involving repeated interaction between rollout generation, reward evaluation, and centralized learning. Distributing rollout execution offers opportunities to leverage more cost-efficient inference resources, but introduces challenges in wide-area coordination and policy dissemination. We present ECHO-2, a distributed RL framework for post-training with remote inference workers and non-negligible dissemination latency. ECHO-2 combines centralized learning with distributed rollouts and treats bounded policy staleness as a user-controlled parameter, enabling rollout generation, dissemination, and training to overlap. We introduce an overlap-based capacity model that relates training time, dissemination latency, and rollout throughput, yielding a practical provisioning rule for sustaining learner utilization. To mitigate dissemination bottlenecks and lower cost, ECHO-2 employs peer-assisted pipelined broadcast and cost-aware activation of heterogeneous workers. Experiments on GRPO post-training of LLMs ranging from 4B to 32B parameters under real wide-area bandwidth regimes show that ECHO-2 significantly improves cost efficiency while preserving RL reward comparable to strong baselines.

تم تقديم ECHO-2 كإطار عمل لتوزيع النشر على نطاق واسع مصمم للتعلم المعزز بتكلفة فعالة، حيث يتناول التحديات المتعلقة بالتنسيق على نطاق واسع ونشر السياسات في التدريب بعد التعليم لنماذج اللغة الكبيرة. يسمح الإطار بتداخل توليد النشر والنشر والتدريب مع اعتبار قدم السياسة المحدودة كمعامل يمكن التحكم فيه من قبل المستخدم.

ECHO-2 se presenta como un marco de despliegue distribuido a gran escala diseñado para el aprendizaje por refuerzo rentable, abordando los desafíos de la coordinación a gran escala y la difusión de políticas en el entrenamiento posterior de grandes modelos de lenguaje. El marco permite la superposición de la generación de despliegues, la difusión y el entrenamiento, considerando la antigüedad limitada de las políticas como un parámetro controlado por el usuario.

ECHO-2 est présenté comme un cadre de déploiement distribué à grande échelle conçu pour un apprentissage par renforcement rentable, abordant les défis de la coordination à grande échelle et de la diffusion des politiques dans l'entraînement postérieur des grands modèles de langage. Le cadre permet le chevauchement de la génération de déploiement, de la diffusion et de l'entraînement tout en considérant la fraîcheur limitée des politiques comme un paramètre contrôlé par l'utilisateur.

ECHO-2 is introduced as a large-scale distributed rollout framework designed for cost-efficient reinforcement learning, addressing the challenges of wide-area coordination and policy dissemination in post-training large language models. The framework allows for overlapping rollout generation, dissemination, and training while treating bounded policy staleness as a user-controlled parameter.