Arc Gradient Descent: A Mathematically Derived Reformulation of Gradient Descent with Phase-Aware, User-Controlled Step Dynamics

A recent study has introduced a pruned version of Capsule Networks (CapsNets), demonstrating that it can operate up to 9.90 times faster than traditional architectures by eliminating 95% of Primary Capsules while maintaining accuracy across various datasets, including MNIST and CIFAR-10.

A new paper introduces an innovative dataset quantization method aimed at reducing storage and communication costs for large-scale datasets on resource-constrained edge devices. This approach focuses on compressing individual samples by minimizing intra-sample redundancy while retaining essential features, marking a shift from traditional inter-sample redundancy methods.

A new study presents a hybrid generative classification approach aimed at enhancing adversarial robustness in deep learning models. The proposed deep ensemble model integrates a pre-trained discriminative network for feature extraction with a generative classification network, achieving high accuracy and robustness against adversarial attacks without the need for adversarial training. Extensive experiments on CIFAR-10 and CIFAR-100 validate its effectiveness.

A new study proposes a structured initialization method for Vision Transformers (ViTs), aiming to integrate the strong inductive biases of Convolutional Neural Networks (CNNs) without altering the architecture. This approach is designed to enhance performance on small datasets while maintaining scalability as data increases.

The recent study on stochastic approximation with block-coordinate optimal stepsizes introduces adaptive stepsize rules designed to minimize the expected distance from an unknown target point. These rules utilize online estimates of the second moment of the search direction, leading to a new method that competes effectively with the widely used Adam algorithm while requiring less memory and fewer hyper-parameters.

The introduction of Adam with Combinatorial Bandit Sampling (AdamCB) enhances the widely used Adam optimizer by integrating combinatorial bandit techniques, allowing for adaptive sample selection during neural network training. This approach addresses the inefficiencies of treating all data samples equally, leading to improved convergence rates and theoretical guarantees over previous methods.

A recent study highlights the vulnerabilities of backdoor defenses in neural networks when subjected to post-training quantization, revealing that INT8 quantization leads to a 0% detection rate for all evaluated defenses while attack success rates remain above 99%. This raises concerns about the effectiveness of existing security measures in machine learning systems.

The introduction of the Folded Optimizer with Approximate Moment (FOAM) presents a new approach to training large language models (LLMs) by compressing optimizer states through block-wise gradient means and a residual correction mechanism. This method aims to alleviate memory bottlenecks associated with traditional optimizers like Adam, which are often memory-intensive during training.