arXiv:2601.22108v2 Announce Type: replace 
Abstract: Continued pretraining is optimized with fixed self-supervised tasks but selected by downstream performance, creating a coarse feedback loop in which practitioners evaluate checkpoints, change data mixtures or objectives, and restart runs, while individual updates remain blind to target capabilities. We ask whether a small set of verifiable downstream examples can provide step-level feedback without directly supervising the learner. We introduce V-pretraining, which decouples a learner trained only with a self-supervised loss from a lightweight task designer that constructs targets or views for unlabeled batches. Given the current learner and batch, V-pretraining scores a candidate construction by predicting the first-order reduction in downstream loss after the induced self-supervised update. The designer maximizes this value; the learner then applies the update with targets or views detached, so downstream labels never update learner parameters. We instantiate V-pretraining as adaptive top-K soft targets for language modeling and learned views or masks for self-supervised vision. Across both modalities, V-pretraining improves target capabilities without degrading generalization. Under wall-clock-matched continued pretraining, it improves GSM8K Pass@1 for Qwen models using 1,024 GSM8K examples only as feedback, including a +7.4 point single-run gain for Qwen2.5-0.5B. In vision, it improves DINOv3 transfer to ADE20K semantic segmentation and NYUv2 depth estimation while preserving ImageNet linear accuracy, suggesting that feedback-guided task construction can improve target capabilities without collapsing general-purpose representations.

تم تقديم نهج جديد للتدريب المستمر، يسمى V-pretraining، والذي يفصل المتعلم عن مصمم المهام، مما يسمح بتغذية راجعة أكثر فعالية تعتمد على الأداء في المهام اللاحقة دون إشراف مباشر. تهدف هذه الطريقة إلى تحسين التعلم الذاتي الإشراف من خلال التنبؤ بانخفاض الخسارة في المهام اللاحقة بعد التحديثات.

Se ha introducido un nuevo enfoque para el preentrenamiento continuo, denominado V-preentrenamiento, que separa al aprendiz del diseñador de tareas, permitiendo una retroalimentación más efectiva basada en el rendimiento en tareas posteriores sin supervisión directa. Este método busca optimizar el aprendizaje auto-supervisado al predecir la reducción de la pérdida en tareas posteriores tras las actualizaciones.

Une nouvelle approche de préentraînement continu, appelée V-préentraînement, a été introduite, séparant l'apprenant du concepteur de tâches, permettant un retour d'information plus efficace basé sur la performance en aval sans supervision directe. Cette méthode vise à optimiser l'apprentissage auto-supervisé en prédisant la réduction de la perte en aval après les mises à jour.

A new approach to continued pretraining, termed V-pretraining, has been introduced, which separates the learner from the task designer, allowing for more effective feedback based on downstream performance without direct supervision. This method aims to optimize self-supervised learning by predicting the reduction in downstream loss following updates.