Scalable neural network-based blackbox optimization

CoGraM (Contextual Granular Merging) is a newly introduced optimization method designed to enhance the merging of neural networks without retraining, addressing issues of accuracy and stability that are prevalent in existing methods like Fisher merging. This multi-stage, context-sensitive approach utilizes rollback mechanisms to prevent harmful updates, thereby improving the robustness of the merged network.

The introduction of the Modified Rectified Power Unit (MRePU) activation function addresses critical issues faced by deep Rectified Power Unit (RePU) networks, such as instability during training due to vanishing or exploding values. This new function retains the advantages of differentiability and universal approximation while ensuring stable training conditions, as demonstrated through extensive theoretical analysis and experiments.

A new framework for Learning to Optimize (L2O) has been proposed to address the challenges of solving constrained bilevel control co-design problems, which are often complex and time-sensitive. This framework utilizes modern differentiation techniques to enhance the efficiency of finding solutions to these optimization problems.

A recent study introduces a multi-view Bayesian optimisation approach that enhances the efficiency of Gaussian process models in engineering design by identifying a low-dimensional latent subspace from input and output data. This method utilizes probabilistic partial least squares (PPLS) to improve the scalability of Bayesian optimisation techniques in complex design scenarios.

A recent study has compared two neural network training strategies—parallel and series-parallel training—specifically for simulating nonlinear dynamical systems. The empirical analysis involved five neural network architectures and practical examples, including a pneumatic valve test bench and an industrial robot benchmark. The findings indicate that while series-parallel training is prevalent, parallel training offers superior long-term prediction accuracy.

A new study proposes a mixed precision accumulation strategy for neural network inference, utilizing a componentwise forward error analysis to optimize error propagation in linear layers. This method suggests that the precision of each output component should be inversely proportional to the condition numbers of the weights and activation functions involved, potentially enhancing computational efficiency.

Sparse Autoencoders (SAEs) have been analyzed to determine their effectiveness in uncovering meaningful concepts within neural network representations. A unified framework has been introduced, framing SAEs as solutions to a bilevel optimization problem, which highlights the inherent biases in concept detection based on the structural assumptions of different SAE architectures.

A new paper has been published addressing the computational challenges of Gaussian process models when applied to autocorrelated data, highlighting the risk of temporal overfitting if autocorrelation is ignored. The authors propose modifications to existing fast Gaussian process approximations to work effectively with blocked data, which helps mitigate these issues.