arXiv:2511.07308v3 Announce Type: replace 
Abstract: Understanding the training dynamics of deep neural networks remains a major open problem, with physics-inspired approaches offering promising insights. Building on this perspective, we develop a thermodynamic framework to describe the stationary distributions of stochastic gradient descent (SGD) with weight decay for scale-invariant neural networks, a setting that both reflects practical architectures with normalization layers and permits theoretical analysis. We establish analogies between training hyperparameters (e.g., learning rate, weight decay) and thermodynamic variables such as temperature, pressure, and volume. Starting with a simplified isotropic noise model, we uncover a close correspondence between SGD dynamics and ideal gas behavior, validated through theory and simulation. Extending to training of neural networks, we show that key predictions of the framework, including the behavior of stationary entropy, align closely with experimental observations. This framework provides a principled foundation for interpreting training dynamics and may guide future work on hyperparameter tuning and the design of learning rate schedulers.

دراسة حديثة تستكشف ديناميات تدريب الشبكات العصبية الثابتة الحجم من خلال إطار حراري، موضحة أوجه التشابه بين الانحدار العشوائي (SGD) وسلوك الغاز المثالي. يهدف هذا النهج إلى تعزيز الفهم حول كيفية تأثير المعلمات الفائقة مثل معدل التعلم وتناقص الوزن على نتائج التدريب.

Un estudio reciente explora la dinámica de entrenamiento de redes neuronales invariantes a escala a través de un marco termodinámico, estableciendo paralelismos entre el descenso de gradiente estocástico (SGD) y el comportamiento de gases ideales. Este enfoque busca mejorar la comprensión de cómo los hiperparámetros como la tasa de aprendizaje y la disminución de peso influyen en los resultados del entrenamiento.

Une étude récente examine la dynamique d'entraînement des réseaux de neurones invariants à l'échelle à travers un cadre thermodynamique, établissant des parallèles entre la descente de gradient stochastique (SGD) et le comportement des gaz idéaux. Cette approche vise à améliorer la compréhension de l'influence des hyperparamètres tels que le taux d'apprentissage et la décroissance du poids sur les résultats d'entraînement.

A recent study explores the training dynamics of scale-invariant neural networks through a thermodynamic framework, drawing parallels between stochastic gradient descent (SGD) and ideal gas behavior. This approach aims to enhance understanding of how hyperparameters like learning rate and weight decay influence training outcomes.