arXiv:2605.29152v1 Announce Type: cross 
Abstract: Randomly initialized neural networks induce a prior over functions, but the predictor used in practice is produced only after training. We ask how much of this initial bias survives the training pipeline. To make the question measurable, we introduce initialization memory: the dependence of the validation-selected predictor on the scale of the random initialization. We perform controlled CIFAR-10 experiments on ResNets where initialization memory already sharply separates training regimes. Low-learning-rate SGD can interpolate while still remembering its initialization: on ResNet-9 with batch size $b=128$, test accuracy varies by $26.5$ percentage points across initialization scales despite $\ge99.5\%$ training accuracy. This is not undertraining: extending the same low-learning-rate regime to $5{,}000$ epochs leaves the spread essentially unchanged. In contrast, Adam-family methods largely erase the dependence. SGD can also be made to forget when larger learning rates are paired with explicit $L_2$ norm control. We interpret these findings in terms of the time scale of forgetting: gradient-flow-like dynamics can preserve initialization memory, whereas stochastic finite-step effects, explicit norm decay, and adaptive preconditioning erase it on scales governed by the size of explicit or implicit regularization. The practical inductive bias of a trained network is therefore not the architectural prior alone, but the architectural prior after being filtered by the forgetting dynamics of the training pipeline; and the same regularizers that improve generalization are precisely those that erase memory of initialization.

دراسة حديثة تبحث في مدى احتفاظ الشبكات العصبية التي تم إعدادها عشوائيًا بتحيزاتها الأولية بعد التدريب، مقدمةً مفهوم ذاكرة الإعداد. تكشف التجارب التي أجريت على هياكل ResNet باستخدام مجموعة بيانات CIFAR-10 عن تباينات كبيرة في دقة الاختبار بناءً على مقاييس الإعداد، خاصةً في ظروف SGD ذات معدل التعلم المنخفض.

Un estudio reciente investiga hasta qué punto las redes neuronales inicializadas aleatoriamente retienen sus sesgos iniciales después del entrenamiento, introduciendo el concepto de memoria de inicialización. Los experimentos realizados en arquitecturas ResNet utilizando el conjunto de datos CIFAR-10 revelan variaciones significativas en la precisión de las pruebas según las escalas de inicialización, especialmente en condiciones de SGD de bajo aprendizaje.

Une étude récente examine dans quelle mesure les réseaux de neurones initialisés aléatoirement conservent leurs biais initiaux après l'entraînement, introduisant le concept de mémoire d'initialisation. Des expériences menées sur des architectures ResNet utilisant le jeu de données CIFAR-10 révèlent des variations significatives de la précision des tests en fonction des échelles d'initialisation, en particulier dans des conditions de SGD à faible taux d'apprentissage.

A recent study investigates the extent to which randomly initialized neural networks retain their initial biases after training, introducing the concept of initialization memory. Experiments conducted on ResNet architectures using the CIFAR-10 dataset reveal significant variations in test accuracy based on initialization scales, particularly under low-learning-rate SGD conditions.