Overparameterized neural networks: Feature learning precedes overfitting, research finds

A recent study has proposed that recurrent neural networks (RNNs) perform computations by dynamically warping their representations of task variables. This hypothesis is supported by a newly developed Riemannian geometric framework that characterizes the manifold topology and geometry of RNNs based on their input data, shedding light on the time-varying geometry of these networks.

A recent study published on arXiv explores continuous-time reinforcement learning (RL) for optimal switching across multiple regimes, utilizing an exploratory formulation with entropy regularization. The research establishes the well-posedness of Hamilton-Jacobi-Bellman equations and characterizes the optimal policy, demonstrating convergence of policy iterations and value functions between exploratory and classical formulations.

A recent study published on arXiv investigates the capabilities of deep linear neural networks in solving underdetermined linear inverse problems, specifically focusing on their convergence when trained using gradient descent with weight decay regularization. The findings suggest that these networks can adapt to unknown low-dimensional structures in the source signal, providing a theoretical basis for their empirical success in machine learning applications.

A recent study published on arXiv presents a significant result linking convex n-widths to covering numbers, particularly in the context of neural networks. This research addresses the challenges of approximating high-dimensional function classes using a limited number of basis functions, revealing that certain classes can be effectively approximated despite the complexities of high-dimensional spaces.

A recent study has applied Singular Learning Theory (SLT), a framework inspired by physics, to analyze grokking and other phase transitions in neural networks. The research empirically investigates SLT's free energy and local learning coefficients, revealing insights into the behavior of neural networks under various conditions.

CoGraM (Contextual Granular Merging) is a newly introduced optimization method designed to enhance the merging of neural networks without retraining, addressing issues of accuracy and stability that are prevalent in existing methods like Fisher merging. This multi-stage, context-sensitive approach utilizes rollback mechanisms to prevent harmful updates, thereby improving the robustness of the merged network.

The introduction of the Modified Rectified Power Unit (MRePU) activation function addresses critical issues faced by deep Rectified Power Unit (RePU) networks, such as instability during training due to vanishing or exploding values. This new function retains the advantages of differentiability and universal approximation while ensuring stable training conditions, as demonstrated through extensive theoretical analysis and experiments.

A new framework for Learning to Optimize (L2O) has been proposed to address the challenges of solving constrained bilevel control co-design problems, which are often complex and time-sensitive. This framework utilizes modern differentiation techniques to enhance the efficiency of finding solutions to these optimization problems.