arXiv:2605.29495v1 Announce Type: new 
Abstract: Continual supervised fine-tuning (SFT) is the de facto recipe for adapting large language models (LLMs) to a stream of downstream tasks, but it suffers from catastrophic forgetting of earlier capabilities. Recent work shows that on-policy signals -- training on the model's own outputs -- reduce forgetting more reliably than off-policy supervision. Existing on-policy methods route this signal through a new training objective (e.g., self-distillation losses with a teacher copy), inheriting an extra forward pass, schedule sensitivity, and stylistic drift from the teacher.We instead route the on-policy signal through the training data source. Our method, On-Policy Replay (OPR), rolls out the most recent checkpoint on a small budget of historical prompts, filters the generations by a task reward, and replays the surviving (prompt, model response) pairs as ordinary SFT examples. There is no teacher, no auxiliary loss, and no on-the-fly distillation. Across three 7--8B instruction-tuned backbones (Qwen2.5-7B-Instruct, Qwen3-8B, Llama3.1-8B-Instruct) on the TRACE continual-learning benchmark, OPR consistently reduces forgetting; on the sharpest stress test (Qwen2.5-7B-Instruct, Sequential SFT BWT -13.93), OPR lifts BWT to -0.65 at a 10% replay budget and to -2.29 at a 1% budget -- a 46% reduction in |BWT| over a tuned Vanilla Replay baseline, with 42--46% reductions observed across all three backbones. We give a KL-shrinkage interpretation that places OPR and prior on-policy distillation methods on a single axis, and we present a counterintuitive finding that explains why Vanilla Replay is already a strong baseline: low-score replay is uniformly worse than Vanilla Replay, demonstrating that the active ingredient in OPR is the on-policy distribution, not the response quality alone.Our code is available at https://github.com/Yancey2024/OnPolicyReplay.

تم تقديم طريقة جديدة تُدعى On-Policy Replay (OPR) للتدريب المستمر تحت الإشراف (SFT) لنماذج اللغة الكبيرة (LLMs)، حيث تعالج مشكلة النسيان الكارثي. تستخدم هذه الطريقة مخرجات النموذج الخاصة لتحسين التدريب دون الحاجة إلى معلم أو خسائر إضافية، مما يبسط العملية ويعزز الأداء المحتمل في المهام اللاحقة.

Se ha introducido un nuevo método llamado On-Policy Replay (OPR) para el ajuste fino supervisado continuo (SFT) de grandes modelos de lenguaje (LLMs), abordando el problema del olvido catastrófico. Este enfoque utiliza las propias salidas del modelo para mejorar el entrenamiento sin necesidad de un maestro o pérdidas auxiliares, simplificando el proceso y potencialmente mejorando el rendimiento en tareas posteriores.

Une nouvelle méthode appelée On-Policy Replay (OPR) a été introduite pour le fine-tuning supervisé continu (SFT) des grands modèles de langage (LLM), abordant le problème de l'oubli catastrophique. Cette approche utilise les propres sorties du modèle pour améliorer l'entraînement sans avoir besoin d'un enseignant ou de pertes auxiliaires, simplifiant ainsi le processus et améliorant potentiellement les performances sur les tâches en aval.

A new method called On-Policy Replay (OPR) has been introduced for continual supervised fine-tuning (SFT) of large language models (LLMs), addressing the issue of catastrophic forgetting. This approach utilizes the model's own outputs to enhance training without the need for a teacher or auxiliary loss, thereby simplifying the process and potentially improving performance on downstream tasks.