arXiv:2412.09498v3 Announce Type: replace-cross 
Abstract: Gradient descent is one of the most widely used iterative algorithms in modern statistical learning. However, its precise algorithmic dynamics in high-dimensional settings remain only partially understood, which has limited its broader potential for statistical inference applications.
  This paper provides a precise, non-asymptotic joint distributional characterization of gradient descent iterates and their debiased statistics in a broad class of empirical risk minimization problems, in the so-called mean-field regime where the sample size is proportional to the signal dimension. Our non-asymptotic state evolution theory holds for both general non-convex loss functions and non-Gaussian data, and reveals the central role of two Onsager correction matrices that precisely characterize the non-trivial dependence among all gradient descent iterates in the mean-field regime.
  Leveraging the joint state evolution characterization, we show that the gradient descent iterate retrieves approximate normality after a debiasing correction via a linear combination of all past iterates, where the debiasing coefficients can be estimated by the proposed gradient descent inference algorithm. This leads to a new algorithmic statistical inference framework based on debiased gradient descent, which (i) applies to a broad class of models with both convex and non-convex losses, (ii) remains valid at each iteration without requiring algorithmic convergence, and (iii) exhibits a certain robustness to possible model misspecification. As a by-product, our framework also provides algorithmic estimates of the generalization error at each iteration. As canonical examples, we demonstrate our theory and inference methods in the single-index regression model and a generalized logistic regression model, where the natural loss functions may exhibit arbitrarily non-convex landscapes.

يُعتبر الانحدار التدرجي خوارزمية تكرارية تُستخدم على نطاق واسع في التعلم الإحصائي، إلا أن دينامياتها في الإعدادات عالية الأبعاد لا تزال غير مفهومة تمامًا. يقدم هذا البحث توصيفًا مشتركًا غير حدّي لتكرارات الانحدار التدرجي وإحصائياتها المُعالجة في مشاكل تقليل المخاطر التجريبية، لا سيما في نظام الحقل المتوسط. تسلط النتائج الضوء على الدور المركزي لمصفوفات تصحيح أونساجر في فهم الاعتماديات بين جميع تكرارات الانحدار التدرجي، القابلة للتطبيق على كل من دوال الخسارة غير المحدبة والبيانات غير الغاوسية.

El descenso de gradiente es un algoritmo iterativo ampliamente utilizado en el aprendizaje estadístico, pero su dinámica en entornos de alta dimensión no se comprende completamente. Este artículo proporciona una caracterización no asintótica de los iterados del descenso de gradiente y sus estadísticas desbiaseadas en problemas de minimización del riesgo empírico, especialmente en el régimen de campo medio. Los hallazgos destacan el papel central de las matrices de corrección de Onsager en la comprensión de las dependencias entre los iterados del descenso de gradiente, aplicables tanto a funcio…

La descente de gradient est un algorithme itératif largement utilisé dans l'apprentissage statistique, mais sa dynamique dans des contextes de haute dimension n'est pas entièrement comprise. Cet article présente une caractérisation non asymptotique des itérations de la descente de gradient et de leurs statistiques débiaisées dans des problèmes de minimisation du risque empirique, en particulier dans le régime de champ moyen. Les résultats mettent en évidence l'importance des matrices de correction d'Onsager pour comprendre les dépendances entre les itérations de la descente de gradient, applic…

Gradient descent is a widely used iterative algorithm in statistical learning, yet its dynamics in high-dimensional settings are not fully understood. This paper presents a non-asymptotic characterization of gradient descent iterates and their debiased statistics in empirical risk minimization problems, particularly in the mean-field regime. The findings highlight the significance of Onsager correction matrices in understanding the dependencies among gradient descent iterates, applicable to both non-convex loss functions and non-Gaussian data.

Gradient descent inference in empirical risk minimization

arXiv:2506.00182v2 Announce Type: replace 
Abstract: Will further scaling up of machine learning models continue to bring success? A significant challenge in answering this question lies in understanding generalization gap, which is the impact of overfitting. Understanding generalization gap behavior of increasingly large-scale machine learning models remains a significant area of investigation, as conventional analyses often link error bounds to model complexity, failing to fully explain the success of extremely large architectures. This research introduces a novel perspective by establishing a model-independent upper bound for generalization gap applicable to algorithms whose outputs are determined solely by the data's histogram, such as empirical risk minimization or gradient-based methods. Crucially, this bound is shown to depend only on the R\'enyi entropy of the data-generating distribution, suggesting that a small generalization gap can be maintained even with arbitrarily large models, provided the data quantity is sufficient relative to this entropy. This framework offers a direct explanation for the phenomenon where generalization performance degrades significantly upon injecting random noise into data, where the performance degrade is attributed to the consequent increase in the data distribution's R\'enyi entropy. Furthermore, we adapt the no-free-lunch theorem to be data-distribution-dependent, demonstrating that an amount of data corresponding to the R\'enyi entropy is indeed essential for successful learning, thereby highlighting the tightness of our proposed generalization bound.

قدمت دراسة حديثة حدًا علويًا مستقلًا عن النموذج لفجوة التعميم في التعلم الآلي، مع التركيز على دور إنترابيا ريني. تتناول هذه البحث القيود المفروضة على التحليلات التقليدية التي تربط حدود الخطأ بتعقيد النموذج، خاصة مع زيادة حجم نماذج التعلم الآلي. تشير النتائج إلى أنه يمكن الحفاظ على فجوة تعميم صغيرة حتى مع الهياكل الكبيرة، وهو أمر حاسم لمستقبل تطبيقات التعلم الآلي.

Un estudio reciente ha introducido un límite superior independiente del modelo para la brecha de generalización en el aprendizaje automático, centrándose en el papel de la entropía de R'enyi. Esta investigación aborda las limitaciones de los análisis tradicionales que vinculan los límites de error con la complejidad del modelo, especialmente a medida que los modelos de aprendizaje automático se escalan. Los hallazgos sugieren que se puede mantener una pequeña brecha de generalización incluso con arquitecturas grandes, lo que es crucial para el futuro de las aplicaciones de aprendizaje automáti…

Une étude récente a introduit une limite supérieure indépendante du modèle pour l'écart de généralisation en apprentissage automatique, en mettant l'accent sur le rôle de l'entropie de R'enyi. Cette recherche aborde les limitations des analyses traditionnelles qui lient les bornes d'erreur à la complexité du modèle, en particulier à mesure que les modèles d'apprentissage automatique se développent. Les résultats suggèrent qu'un petit écart de généralisation peut être maintenu même avec de grandes architectures, ce qui est crucial pour l'avenir des applications d'apprentissage automatique.

A recent study has introduced a model-independent upper bound for the generalization gap in machine learning, focusing on the role of R'enyi entropy. This research addresses the limitations of traditional analyses that link error bounds to model complexity, particularly as machine learning models scale up. The findings suggest that a small generalization gap can be maintained even with large architectures, which is crucial for the future of machine learning applications.

Gradient descent inference in empirical risk minimization

Was this article worth reading? Share it

Aqaba.ai

Dyad

AINF Membership