arXiv:2601.08253v1 Announce Type: new 
Abstract: We analyze initialization dynamics for LDLT-based $\mathcal{L}$-Lipschitz layers by deriving the exact marginal output variance when the underlying parameter matrix $W_0\in \mathbb{R}^{m\times n}$ is initialized with IID Gaussian entries $\mathcal{N}(0,\sigma^2)$. The Wishart distribution, $S=W_0W_0^\top\sim\mathcal{W}_m(n,\sigma^2 \boldsymbol{I}_m)$, used for computing the output marginal variance is derived in closed form using expectations of zonal polynomials via James' theorem and a Laplace-integral expansion of $(\alpha \boldsymbol{I}_m+S)^{-1}$. We develop an Isserlis/Wick-based combinatorial expansion for $\operatorname{\mathbb{E}}\left[\operatorname{tr}(S^k)\right]$ and provide explicit truncated moments up to $k=10$, which yield accurate series approximations for small-to-moderate $\sigma^2$. Monte Carlo experiments confirm the theoretical estimates. Furthermore, empirical analysis was performed to quantify that, using current He or Kaiming initialization with scaling $1/\sqrt{n}$, the output variance is $0.41$, whereas the new parameterization with $10/ \sqrt{n}$ for $\alpha=1$ results in an output variance of $0.9$. The findings clarify why deep $\mathcal{L}$-Lipschitz networks suffer rapid information loss at initialization and offer practical prescriptions for choosing initialization hyperparameters to mitigate this effect. However, using the Higgs boson classification dataset, a hyperparameter sweep over optimizers, initialization scale, and depth was conducted to validate the results on real-world data, showing that although the derivation ensures variance preservation, empirical results indicate He initialization still performs better.

الدراسة الحديثة حول طبقات L-Lipschitz المستندة إلى LDLT تقدم تحليلًا مفصلًا لديناميات التهيئة، حيث تستخرج التباين الهامشي الناتج بدقة عندما يتم تهيئة مصفوفة المعلمات بإدخالات غاوسية IID. تستفيد النتائج من توزيع ويشارت وتستخدم تقنيات رياضية متقدمة لتقديم تعبيرات مغلقة لشروط حساب التباين.

El estudio reciente sobre capas L-Lipschitz basadas en LDLT presenta un análisis detallado de la dinámica de inicialización, derivando la varianza marginal de salida exacta cuando la matriz de parámetros se inicializa con entradas gaussianas IID. Los hallazgos aprovechan la distribución de Wishart y emplean técnicas matemáticas avanzadas para proporcionar expresiones en forma cerrada para los cálculos de varianza.

La récente étude sur les couches L-Lipschitz basées sur LDLT présente une analyse détaillée des dynamiques d'initialisation, dérivant la variance marginale de sortie exacte lorsque la matrice de paramètres est initialisée avec des entrées gaussiennes IID. Les résultats s'appuient sur la distribution de Wishart et utilisent des techniques mathématiques avancées pour fournir des expressions en forme fermée pour les calculs de variance.

The recent study on LDLT-based L-Lipschitz layers presents a detailed analysis of initialization dynamics, deriving the exact marginal output variance when the parameter matrix is initialized with IID Gaussian entries. The findings leverage the Wishart distribution and employ advanced mathematical techniques to provide closed-form expressions for variance calculations.

LDLT L-Lipschitz Network Weight Parameterization Initialization

arXiv:2509.13628v3 Announce Type: replace-cross 
Abstract: We study trade-offs between convergence rate and robustness to gradient errors in the context of first-order methods. Our focus is on generalized momentum methods (GMMs)--a broad class that includes Nesterov's accelerated gradient, heavy-ball, and gradient descent methods--for minimizing smooth strongly convex objectives. We allow stochastic gradient errors that may be adversarial and biased, and quantify robustness of these methods to gradient errors via the risk-sensitive index (RSI) from robust control theory. For quadratic objectives with i.i.d. Gaussian noise, we give closed form expressions for RSI in terms of solutions to 2x2 matrix Riccati equations, revealing a Pareto frontier between RSI and convergence rate over the choice of step-size and momentum parameters. We then prove a large-deviation principle for time-averaged suboptimality in the large iteration limit and show that the rate function is, up to a scaling, the convex conjugate of the RSI function. We further show that the rate function and RSI are linked to the $H_\infty$-norm--a measure of robustness to the worst-case deterministic gradient errors--so that stronger worst-case robustness (smaller $H_\infty$-norm) leads to sharper decay of the tail probabilities for the average suboptimality. Beyond quadratics, under potentially biased sub-Gaussian gradient errors, we derive non-asymptotic bounds on a finite-time analogue of the RSI, yielding finite-time high-probability guarantees and non-asymptotic large-deviation bounds for the averaged iterates. In the case of smooth strongly convex functions, we also observe an analogous trade-off between RSI and convergence-rate bounds. To our knowledge, these are the first non-asymptotic guarantees for GMMs with biased gradients and the first risk-sensitive analysis of GMMs. Finally, we provide numerical experiments on a robust regression problem to illustrate our results.

تدرس دراسة حديثة التوازن بين معدلات التقارب والقدرة على التحمل أمام أخطاء التدرج في طرق من الدرجة الأولى، مع التركيز بشكل خاص على طرق الزخم العامة (GMM) لتقليل الأهداف المحدبة السلسة والقوية. تقوم الدراسة بتحديد مدى قدرة هذه الطرق على تحمل أخطاء التدرج العشوائية باستخدام مؤشر الحساسية للمخاطر (RSI) من نظرية التحكم القوي، وتقدم تعبيرات مغلقة الشكل لـ RSI في سياقات معينة.

Un estudio reciente explora los compromisos entre las tasas de convergencia y la robustez frente a errores de gradiente en métodos de primer orden, centrándose particularmente en métodos de momento generalizados (GMM) para minimizar objetivos convexos suaves y fuertemente convexos. La investigación cuantifica la robustez de estos métodos ante errores de gradiente estocásticos utilizando el índice de sensibilidad al riesgo (RSI) de la teoría del control robusto, proporcionando expresiones en forma cerrada para el RSI en contextos específicos.

Une étude récente examine les compromis entre les taux de convergence et la robustesse face aux erreurs de gradient dans les méthodes de premier ordre, en se concentrant particulièrement sur les méthodes de moment généralisées (GMM) pour minimiser des objectifs convexes lisses et fortement convexes. La recherche quantifie la robustesse de ces méthodes face aux erreurs de gradient stochastiques en utilisant l'indice de sensibilité au risque (RSI) de la théorie du contrôle robuste, fournissant des expressions sous forme fermée pour le RSI dans des contextes spécifiques.

A recent study explores the trade-offs between convergence rates and robustness to gradient errors in first-order methods, particularly focusing on generalized momentum methods (GMMs) for minimizing smooth strongly convex objectives. The research quantifies the robustness of these methods to stochastic gradient errors using the risk-sensitive index (RSI) from robust control theory, providing closed-form expressions for RSI in specific contexts.

LDLT L-Lipschitz Network Weight Parameterization Initialization

Was this article worth reading? Share it

LucidQuery AI

LangWatch

Langfuse

Langtail

Hypertune

FastML

Ready to build your own newsroom?