arXiv:2604.09258v2 Announce Type: replace 
Abstract: The foundational capabilities of large language models are acquired during pretraining on internet-scale, highly heterogeneous data mixtures. In this work, we investigate an interesting geometric question regarding the converged state of pretraining: Does the model converge to a common minimizer across all data sources (e.g., \cref{fig:cwa_illustration:close}), or merely a minimizer of the summed loss (e.g., \cref{fig:cwa_illustration:distant})? We hypothesize that the geometric "closeness" of task-specific minima is intrinsically linked to downstream generalization. We reveal that standard optimizers (e.g., AdamW) often converge to points where task-specific minima are distant from each other. To address this, we propose the Nexus optimizer, which encourages the closeness of these minima by maximizing gradient similarity during optimization. Experiments across models ranging from 130M to 3B parameters, various data mixtures and hyperparameter schedules, show that Nexus \textit{significantly boosts downstream performance}, despite \textit{achieving the same pretraining loss} (see \cref{fig:demo:benchmark}). Notably, on the 3B model, Nexus reduces the out-of-distribution loss by 0.012 and yields up to a 15.0\% accuracy improvement on complex reasoning tasks (e.g., GSM8k). This finding challenges the reliance on pretraining loss as the sole proxy for model evaluation and demonstrates the importance of implicit biases in unlocking downstream generalization.

دراسة جديدة بعنوان 'Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima' تستكشف سلوك التقارب لنماذج اللغة الكبيرة أثناء التدريب المسبق. تسلط الدراسة الضوء على أن المحسنات القياسية مثل AdamW غالبًا ما تؤدي إلى نقاط حد أدنى محددة للمهام تكون بعيدة عن بعضها البعض، مما قد يعيق التعميم في المهام اللاحقة. لمواجهة ذلك، يقترح المؤلفون المحسن Nexus، المصمم لتعزيز قرب هذه النقاط الدنيا من خلال زيادة تشابه التدرجات أثناء التحسين.

Un nuevo estudio titulado 'Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima' investiga el comportamiento de convergencia de los grandes modelos de lenguaje durante el preentrenamiento. La investigación destaca que los optimizadores estándar como AdamW a menudo conducen a mínimos específicos de tareas distantes, lo que puede obstaculizar la generalización en tareas posteriores. Para abordar esto, los autores proponen el optimizador Nexus, diseñado para mejorar la cercanía de estos mínimos maximizando la similitud del gradiente durante la optimización.

Une nouvelle étude intitulée 'Nexus : Same Pretraining Loss, Better Downstream Generalization via Common Minima' examine le comportement de convergence des grands modèles de langage lors du préentraînement. La recherche souligne que les optimisateurs standards comme AdamW conduisent souvent à des minima spécifiques aux tâches éloignés, ce qui peut entraver la généralisation en aval. Pour y remédier, les auteurs proposent l'optimiseur Nexus, conçu pour améliorer la proximité de ces minima en maximisant la similarité des gradients pendant l'optimisation.

A new study titled 'Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima' explores the convergence behavior of large language models during pretraining. The research highlights that standard optimizers like AdamW often lead to distant task-specific minima, which may hinder downstream generalization. To counter this, the authors propose the Nexus optimizer, designed to enhance the closeness of these minima by maximizing gradient similarity during optimization.

Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima

arXiv:2606.13894v1 Announce Type: new 
Abstract: AdamW is a default optimizer for modern deep learning, but its first and second moment states add roughly two parameter-sized buffers to training memory. We propose Gefen, a memory-efficient optimizer that automatically shares second-moment estimates across parameter blocks and quantizes the first moment using a learned codebook, thereby reducing AdamW's memory footprint by ~8x while maintaining the same performance, corresponding to a reduction of 6.5 GiB per billion parameters. The method is motivated by a theoretical result showing that large mixed Hessian entries constrain the ratio of squared gradients toward one, suggesting that Hessian-aligned parameters are natural candidates for sharing second-moment statistics. Since computing Hessians is impractical at scale, Gefen infers block structure from the initial squared gradients, requiring no architecture-specific metadata or hyperparameters beyond AdamW defaults. Gefen learns an exact histogram-based dynamic-programming quantization codebook and reuses the same blocks for first-moment scaling. Across diverse experiments, Gefen achieves the lowest peak optimizer memory among the compared AdamW-like methods while maintaining AdamW-level performance. In FSDP and DDP training, the reduced memory footprint enables larger microbatches and improves throughput significantly over AdamW, providing a practical drop-in replacement with lower memory usage that can increase throughput and enable training larger models or using larger batch sizes. We provide the complete Python implementation, including fused CUDA kernels at https://github.com/ndvbd/Gefen

تم تقديم Gefen، وهو مُحسّن جديد موفر للذاكرة، كبديل لـ AdamW، مما يقلل بشكل كبير من استخدام الذاكرة أثناء تدريب التعلم العميق بحوالي 8 مرات مع الحفاظ على الأداء. يتم تحقيق ذلك من خلال مشاركة تقديرات اللحظة الثانية تلقائيًا وتكميم اللحظة الأولى.

Gefen, un nuevo optimizador eficiente en memoria, ha sido presentado como una alternativa a AdamW, reduciendo significativamente el uso de memoria durante el entrenamiento de aprendizaje profundo en aproximadamente 8x, manteniendo el rendimiento. Esto se logra mediante el intercambio automático de estimaciones del segundo momento y la cuantización del primer momento.

Gefen, un nouvel optimiseur économe en mémoire, a été introduit comme alternative à AdamW, réduisant considérablement l'utilisation de la mémoire lors de l'entraînement en apprentissage profond d'environ 8x tout en maintenant la performance. Cela est réalisé grâce au partage automatique des estimations du second moment et à la quantification du premier moment.

Gefen, a new memory-efficient optimizer, has been introduced as an alternative to AdamW, significantly reducing memory usage during deep learning training by approximately 8x while maintaining performance. This is achieved through automatic sharing of second-moment estimates and quantization of the first moment.

Gefen: Optimized Stochastic Optimizer

arXiv:2606.13657v2 Announce Type: replace 
Abstract: On-policy distillation (\textsc{OPD}) has recently become a prominent post-training recipe by combining two desirable ingredients: on-policy student trajectories and dense teacher supervision. However, how this hybrid changes a model's parameters remains unclear. Across several language and vision-language model pairs and \textsc{OPD} use cases, our analysis yields two main findings. On sparsity, \textsc{OPD} updates are small and coordinate-sparse. They are distributed across layers, with the largest relative movement usually appearing in FFN modules. This sparse structure is operationally useful: training only the discovered subnetwork nearly recovers full-training performance. The sparse support does not remove the need for adaptive optimization: SGD, previously reported to be competitive in \textsc{RLVR}, underperforms AdamW in our \textsc{OPD} optimizer ablation, suggesting that dense teacher supervision preserves useful momentum structure and heterogeneous second-moment scales. On geometry, the updates are numerically full-rank but spectrally concentrated; they lie mostly away from the principal singular subspaces of the source weights and fall disproportionately on coordinates where the source weights are close to zero. These findings suggest that dense teacher supervision does not turn \textsc{OPD} into ordinary dense parameter rewriting; instead, \textsc{OPD} retains important geometric signatures of on-policy post-training.

أظهرت دراسة حديثة حول التقطير في السياسة (OPD) أن هذه الطريقة تجمع بين مسارات الطلاب في السياسة وإشراف المعلمين الكثيف، مما يؤدي إلى تحديثات صغيرة وقليلة التنسيق في معلمات النموذج. تشير التحليلات إلى أن التغييرات الأكثر أهمية تحدث في وحدات الشبكة العصبية الأمامية، مما يشير إلى نهج منظم لتحسين النموذج.

Un estudio reciente sobre la destilación en política (OPD) revela que este método combina trayectorias de estudiantes en política con supervisión densa de maestros, lo que lleva a actualizaciones pequeñas y escasamente coordinadas en los parámetros del modelo. El análisis indica que los cambios más significativos ocurren en los módulos de red neuronal de retroalimentación, sugiriendo un enfoque estructurado para la optimización del modelo.

Une étude récente sur la distillation en politique (OPD) révèle que cette méthode combine des trajectoires d'étudiants en politique avec une supervision dense des enseignants, entraînant des mises à jour petites et coordonnées dans les paramètres du modèle. L'analyse indique que les changements les plus significatifs se produisent dans les modules de réseau de neurones à propagation avant, suggérant une approche structurée de l'optimisation des modèles.

A recent study on on-policy distillation (OPD) reveals that this method combines on-policy student trajectories with dense teacher supervision, leading to small and coordinate-sparse updates in model parameters. The analysis indicates that the most significant changes occur in feedforward neural network modules, suggesting a structured approach to model optimization.

Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima

Was this article worth reading? Share it

Ready to build your own newsroom?