arXiv:2506.06501v2 Announce Type: replace-cross 
Abstract: We study realizable continual linear regression under random task orderings, a common setting for developing continual learning theory. In this setup, the worst-case expected loss after $k$ learning iterations admits a lower bound of $\Omega(1/k)$. However, prior work using an unregularized scheme has only established an upper bound of $O(1/k^{1/4})$, leaving a significant gap. Our paper proves that this gap can be narrowed, or even closed, using two frequently used regularization schemes: (1) explicit isotropic $\ell_2$ regularization, and (2) implicit regularization via finite step budgets. We show that these approaches, which are used in practice to mitigate forgetting, reduce to stochastic gradient descent (SGD) on carefully defined surrogate losses. Through this lens, we identify a fixed regularization strength that yields a near-optimal rate of $O(\log k / k)$. Moreover, formalizing and analyzing a generalized variant of SGD for time-varying functions, we derive an increasing regularization strength schedule that provably achieves an optimal rate of $O(1/k)$. This suggests that schedules that increase the regularization coefficient or decrease the number of steps per task are beneficial, at least in the worst case.

أظهرت دراسة حديثة حول الانحدار الخطي المستمر تقدمًا كبيرًا في تقليل الفجوة بين حدود الخسارة المتوقعة، وهو أمر حاسم لتقدم نظرية التعلم المستمر. من خلال معالجة قيود الأساليب غير المنتظمة السابقة، لا تثبت هذه الأبحاث فقط وجود حد أدنى قدره أوميغا(1/k)، بل تُظهر أيضًا إمكانية تحسين الحد الأقصى. هذا العمل مهم لأنه يعزز فهمنا لكيفية تحسين التعلم بمرور الوقت، مما قد يؤدي إلى خوارزميات أكثر كفاءة في التعلم الآلي.

Un estudio reciente sobre la regresión lineal continua ha logrado avances significativos al reducir la brecha entre los límites de pérdida esperada, lo cual es crucial para avanzar en la teoría del aprendizaje continuo. Al abordar las limitaciones de enfoques no regularizados anteriores, esta investigación no solo establece un límite inferior de Omega(1/k), sino que también demuestra que se puede mejorar el límite superior. Este trabajo es importante ya que mejora nuestra comprensión de cómo se puede optimizar el aprendizaje a lo largo del tiempo, lo que podría llevar a algoritmos más eficientes en el aprendizaje automático.

Une étude récente sur la régression linéaire continue a réalisé des avancées significatives en réduisant l'écart entre les bornes de perte attendue, ce qui est crucial pour faire progresser la théorie de l'apprentissage continu. En abordant les limitations des approches non régularisées précédentes, cette recherche établit non seulement une borne inférieure de Omega(1/k), mais démontre également que la borne supérieure peut être améliorée. Ce travail est important car il améliore notre compréhension de la façon dont l'apprentissage peut être optimisé au fil du temps, ce qui pourrait conduire à des algorithmes plus efficaces en apprentissage automatique.

A recent study on continual linear regression has made significant strides in narrowing the gap between expected loss bounds, which is crucial for advancing continual learning theory. By addressing the limitations of previous unregularized approaches, this research not only establishes a lower bound of Omega(1/k) but also demonstrates that the upper bound can be improved. This work is important as it enhances our understanding of how learning can be optimized over time, potentially leading to more efficient algorithms in machine learning.

Optimal Rates in Continual Linear Regression via Increasing Regularization

arXiv:2509.21181v3 Announce Type: replace-cross 
Abstract: For overparameterized linear regression with isotropic Gaussian design and minimum-$\ell_p$ interpolator $p\in(1,2]$, we give a unified, high-probability characterization for the scaling of the family of parameter norms $ \\{ \lVert \widehat{w_p} \rVert_r \\}_{r \in [1,p]} $ with sample size.
  We solve this basic, but unresolved question through a simple dual-ray analysis, which reveals a competition between a signal *spike* and a *bulk* of null coordinates in $X^\top Y$, yielding closed-form predictions for (i) a data-dependent transition $n_\star$ (the "elbow"), and (ii) a universal threshold $r_\star=2(p-1)$ that separates $\lVert \widehat{w_p} \rVert_r$'s which plateau from those that continue to grow with an explicit exponent.
  This unified solution resolves the scaling of *all* $\ell_r$ norms within the family $r\in [1,p]$ under $\ell_p$-biased interpolation, and explains in one picture which norms saturate and which increase as $n$ grows.
  We then study diagonal linear networks (DLNs) trained by gradient descent. By calibrating the initialization scale $\alpha$ to an effective $p_{\mathrm{eff}}(\alpha)$ via the DLN separable potential, we show empirically that DLNs inherit the same elbow/threshold laws, providing a predictive bridge between explicit and implicit bias.
  Given that many generalization proxies depend on $\lVert \widehat {w_p} \rVert_r$, our results suggest that their predictive power will depend sensitively on which $l_r$ norm is used.

قدمت دراسة حديثة توصيفًا موحدًا لتدرج معايير المعلمات في الانحدار الخطي المفرط المعلمات والشبكات الخطية القطرية تحت انحياز $l_p$. يتناول هذا العمل السؤال غير المحسوم حول كيفية تصرف عائلة معايير $l_r$ مع أحجام عينات مختلفة، كاشفًا عن تنافس بين ذروات الإشارة والإحداثيات الفارغة في البيانات.

Un estudio reciente ha proporcionado una caracterización unificada del escalado de las normas de parámetros en la regresión lineal sobreparametrizada y las redes lineales diagonales bajo sesgo $l_p$. Este trabajo aborda la cuestión no resuelta de cómo se comporta la familia de normas $l_r$ con diferentes tamaños de muestra, revelando una competencia entre picos de señal y coordenadas nulas en los datos.

Une étude récente a fourni une caractérisation unifiée du comportement des normes de paramètres dans la régression linéaire surparamétrée et les réseaux linéaires diagonaux sous biais $l_p$. Ce travail aborde la question non résolue de la manière dont la famille des normes $l_r$ se comporte avec des tailles d'échantillon variables, révélant une compétition entre les pics de signal et les coordonnées nulles dans les données.

A recent study has provided a unified characterization of the scaling of parameter norms in overparameterized linear regression and diagonal linear networks under $l_p$ bias. This work addresses the unresolved question of how the family of $l_r$ norms behaves with varying sample sizes, revealing a competition between signal spikes and null coordinates in the data.

Optimal Rates in Continual Linear Regression via Increasing Regularization

Was this article worth reading? Share it

LucidQuery AI

Hypertune

Retentional's Success Planner