arXiv:2606.14259v1 Announce Type: new 
Abstract: Prior work has identified several factors that can contribute to the performance gap between Adam and SGD, spanning data aspects, architecture design, and optimization properties. Yet these explanations are often studied in isolation, leaving their relative importance unclear. In this work, we revisit these hypotheses through a controlled empirical study across vision, language, genomics, and graph tasks, spanning modern and classical architectures, and carefully designed training setups. Our results suggest that no single factor consistently explains the Adam--SGD gap. For instance, the Adam advantage can (1) persist under a uniform vocabulary distribution yet nearly disappear under a heavy-tailed one; (2) reverse in favor of SGD in softmax-attention models; and (3) become larger under soft architectural modifications, e.g., when ReLU is replaced by a GeLU nonlinearity. This suggests that the gap arises from nontrivial data and architecture interactions, rather than from a single common factor. Yet, we observe a pattern across our settings: a \emph{crossover batch size} at which the relative advantage shifts from SGD to Adam as the batch size scales. These empirical results are captured by our theoretical gap model, which predicts this batch-size-dependent crossover. Our perspective helps reconcile several existing hypotheses while offering practical insights across domains.

أظهرت الأبحاث الأخيرة الفجوة في الأداء بين خوارزميات التحسين آدم وSGD، كاشفةً عن أن هناك عوامل متعددة تسهم في هذه الفجوة عبر مهام وهياكل مختلفة. تشير الدراسة إلى أنه لا يمكن تفسير الفجوة بين آدم وSGD بعامل واحد، حيث يمكن أن تختلف مزايا آدم بشكل كبير بناءً على توزيع البيانات والاختيارات المعمارية.

Investigaciones recientes han examinado la brecha de rendimiento entre los algoritmos de optimización Adam y SGD, revelando que múltiples factores contribuyen a esta discrepancia en diversas tareas y arquitecturas. El estudio indica que ninguna explicación única puede dar cuenta de la brecha Adam-SGD, ya que las ventajas de Adam pueden variar significativamente según la distribución de datos y las elecciones arquitectónicas.

Des recherches récentes ont examiné l'écart de performance entre les algorithmes d'optimisation Adam et SGD, révélant que plusieurs facteurs contribuent à cette disparité à travers diverses tâches et architectures. L'étude indique qu'aucune explication unique ne peut rendre compte de l'écart Adam-SGD, car les avantages d'Adam peuvent varier considérablement en fonction de la distribution des données et des choix architecturaux.

Recent research has examined the performance gap between the Adam and SGD optimization algorithms, revealing that multiple factors contribute to this discrepancy across various tasks and architectures. The study indicates that no single explanation can account for the Adam-SGD gap, as the advantages of Adam can vary significantly based on data distribution and architectural choices.