SWAT-NN: Simultaneous Weights and Architecture Training for Neural Networks in a Latent Space

The study focuses on data-driven least squares (LS) problems constrained by semidefinite (SD) conditions, providing finite-sample guarantees on the spectrum of optimal solutions when these constraints are relaxed. A high confidence bound is introduced, allowing for a simpler program to be solved instead of the full SDLS problem, ensuring that the eigenvalues of the solution remain close to those dictated by the SD constraints. The certificate developed is easy to compute and requires independent and identically distributed samples.

A new framework for reconstructing the microstructure of heterogeneous porous materials has been proposed, integrating neural networks with the sliced-Wasserstein metric. This approach enhances microstructure characterization and reconstruction, which are essential for modeling materials in engineering applications. By utilizing local pattern distribution and a controlled sampling strategy, the framework aims to improve the controllability and applicability of microstructure reconstruction, even with small sample sizes.

The study presents the first global convergence result for neural networks using a two-stage least squares (2SLS) approach in nonparametric instrumental variable regression (NPIV). By employing mean-field Langevin dynamics (MFLD) and addressing a bilevel optimization problem, the researchers introduce a novel first-order algorithm named F²BMLD. The findings include convergence and generalization bounds, highlighting a trade-off in the choice of Lagrange multipliers, and the method's effectiveness is validated through offline reinforcement learning experiments.

The article discusses the significance of hyperparameter tuning in ensuring the convergence of machine learning models, particularly through stochastic gradient descent (SGD). It presents a phase diagram of a multilayer neural network, where each phase reflects unique dynamics of singular values in weight matrices. The study draws parallels with disordered systems, interpreting the loss landscape as a disordered feature space, with the initial variance of weight matrices representing disorder strength and temperature linked to the learning rate and batch size.

The article discusses the limitations of programming neural networks directly, highlighting the reliance on indirect learning algorithms like gradient descent. It introduces Cajal, a new higher-order programming language designed to compile algorithms into linear neurons, thus enabling the expression of discrete algorithms in a differentiable manner. This advancement aims to enhance the capabilities of neural networks by overcoming the challenges posed by traditional programming methods.

The article discusses the approximation and learning capabilities of neural networks concerning high-dimensional geometry and statistical learning theory. It examines the impact of the VC dimension on the networks' ability to approximate functions and learn from data samples. While a finite VC dimension is beneficial for uniform convergence of empirical errors, it may hinder function approximation from probability distributions relevant to specific applications. The study highlights the deterministic behavior of approximation and empirical errors in networks with finite VC dimensions.

The article reviews modern optimization methods for training neural networks, focusing on efficiency and scalability. It presents state-of-the-art algorithms within a unified framework, emphasizing the need to adapt to specific problem structures. The content is designed for both practitioners and researchers interested in the latest advancements in this field.

The article presents a new approach to analyzing the Neural Tangent Kernel (NTK) through a matrix-free perspective, utilizing trace estimation techniques. This method allows for rapid computation of the NTK's trace, Frobenius norm, effective rank, and alignment, particularly beneficial for recurrent architectures. The authors demonstrate that one-sided estimators can outperform traditional methods in low-sample scenarios, highlighting the potential for significant speedups in computational efficiency.