Distillation-Guided Structural Transfer for Continual Learning Beyond Sparse Distributed Memory

The Fourier Analysis Network (FAN) has been proposed as a method to enhance neural network performance by integrating sine and cosine functions in place of some ReLU activations. Research indicates that while sine functions contribute positively to performance, cosine functions may hinder it. This study clarifies that the benefits arise from the sine function's local behavior, particularly near zero, which helps address the vanishing-gradient problem.

A new gradient-based membership inference attack for federated learning has been introduced, leveraging the temporal evolution of last-layer gradients across multiple federated rounds. This method does not require access to private datasets and is designed to address both semi-honest and malicious adversaries, expanding the scope of potential data leaks in federated learning scenarios.

A systematic study has been conducted on the privacy-utility relationship in post-training quantization (PTQ) of deep neural networks, focusing on three algorithms: AdaRound, BRECQ, and OBC. The research reveals that low-precision PTQs, specifically at 4-bit, 2-bit, and 1.58-bit levels, can significantly reduce privacy leakage while maintaining model performance across datasets like CIFAR-10, CIFAR-100, and TinyImageNet.

A new study presents REAL (Representation Enhanced Analytic Learning), a method designed to improve exemplar-free class-incremental learning (EFCIL) by addressing issues of representation and knowledge utilization in existing analytic continual learning frameworks. REAL employs a dual-stream pretraining approach followed by a representation-enhancing distillation process to create a more effective classifier during class-incremental learning.

A new framework called Arithmetic-Intensity-Aware Quantization (AIQ) has been introduced to optimize the performance of neural networks by selecting per-layer bit-widths that enhance arithmetic intensity while minimizing accuracy loss. This method has shown a significant increase in throughput and efficiency on models like ResNet-20 and MobileNetV2, outperforming traditional quantization techniques.

A new one-cycle structured pruning framework has been proposed, integrating pre-training, pruning, and fine-tuning into a single training cycle, which aims to enhance efficiency while maintaining accuracy. This method identifies an optimal sub-network early in the training process, utilizing norm-based group saliency criteria and structured sparsity regularization to improve performance.