SeeDNorm: Self-Rescaled Dynamic Normalization

Recent research has revealed that modern neural networks, which are highly overparameterized, can learn underlying features from structured datasets before they begin to overfit, even when exposed to random data. This finding challenges previous assumptions about the limitations of overparameterized models in machine learning.

A recent study has proposed that recurrent neural networks (RNNs) perform computations by dynamically warping their representations of task variables. This hypothesis is supported by a newly developed Riemannian geometric framework that characterizes the manifold topology and geometry of RNNs based on their input data, shedding light on the time-varying geometry of these networks.

A recent study published on arXiv explores continuous-time reinforcement learning (RL) for optimal switching across multiple regimes, utilizing an exploratory formulation with entropy regularization. The research establishes the well-posedness of Hamilton-Jacobi-Bellman equations and characterizes the optimal policy, demonstrating convergence of policy iterations and value functions between exploratory and classical formulations.

A recent study published on arXiv investigates the capabilities of deep linear neural networks in solving underdetermined linear inverse problems, specifically focusing on their convergence when trained using gradient descent with weight decay regularization. The findings suggest that these networks can adapt to unknown low-dimensional structures in the source signal, providing a theoretical basis for their empirical success in machine learning applications.

A recent study published on arXiv presents a significant result linking convex n-widths to covering numbers, particularly in the context of neural networks. This research addresses the challenges of approximating high-dimensional function classes using a limited number of basis functions, revealing that certain classes can be effectively approximated despite the complexities of high-dimensional spaces.

Recent research has introduced spectral gradient methods, including the Muon optimizer, as alternatives to traditional Euclidean gradient descent for training deep neural networks and transformers. A proposed layerwise condition predicts when spectral updates can lead to greater loss reduction compared to Euclidean steps, particularly in specific parameter configurations.

A recent study has applied Singular Learning Theory (SLT), a framework inspired by physics, to analyze grokking and other phase transitions in neural networks. The research empirically investigates SLT's free energy and local learning coefficients, revealing insights into the behavior of neural networks under various conditions.

CoGraM (Contextual Granular Merging) is a newly introduced optimization method designed to enhance the merging of neural networks without retraining, addressing issues of accuracy and stability that are prevalent in existing methods like Fisher merging. This multi-stage, context-sensitive approach utilizes rollback mechanisms to prevent harmful updates, thereby improving the robustness of the merged network.