UNSEEN: Enhancing Dataset Pruning from a Generalization Perspective

The article presents a novel approach to Knowledge Distillation (KD) through the introduction of a Dynamic Temperature Scheduler (DTS). Traditional KD methods utilize a fixed temperature during training, which can be inefficient. The DTS adapts the temperature dynamically based on the cross-entropy loss gap between the teacher and student models. This method allows for softer probabilities in the early training stages and sharper probabilities later on, optimizing the learning process. The DTS has been validated across various KD strategies, including on datasets like CIFAR-100 and Tiny-ImageN…

The Generalized Denoising Diffusion Codebook Models (gDDCM) have been introduced as an extension of the Denoising Diffusion Codebook Models (DDCM). This new model utilizes the Denoising Diffusion Probabilistic Model (DDPM) and enhances image compression by replacing random noise in the backward process with noise sampled from specific sets. The gDDCM is applicable to various diffusion models, including Score-Based Models and Consistency Models. Evaluations on CIFAR-10 and LSUN Bedroom datasets show improved performance over previous methods.

The paper introduces FLARE, an adaptive reputation-based framework designed to enhance client reliability in federated learning (FL). FL addresses the challenge of maintaining data privacy during collaborative model training but is susceptible to threats from malicious clients. FLARE shifts client reliability assessment from binary to a continuous, multi-dimensional evaluation, incorporating performance consistency and adaptive thresholds to improve model integrity against Byzantine attacks and data poisoning.

The article challenges the prevailing belief that noise conditioning is essential for the success of denoising diffusion models. Through an investigation of various denoising-based generative models without noise conditioning, the authors found that most models showed graceful degradation, with some performing better without it. A noise-unconditional model achieved a competitive FID score of 2.23 on CIFAR-10, suggesting that the community should reconsider the foundations of denoising generative models.

The article discusses a new approach to attention mechanisms in artificial intelligence, inspired by biological synaptic plasticity. This method aims to improve energy efficiency in spiking neural networks (SNNs) compared to traditional Transformers, which rely on dot-product similarity. The research highlights the limitations of current spiking attention models and proposes a biologically inspired spiking neuromorphic transformer that could reduce the carbon footprint associated with large language models (LLMs) like GPT.

Deep neural networks are susceptible to adversarial perturbations that can lead to incorrect predictions. The paper introduces DeepDefense, a defense framework utilizing Gradient-Feature Alignment (GFA) regularization across multiple layers to mitigate this vulnerability. By aligning input gradients with internal feature representations, DeepDefense creates a smoother loss landscape, reducing sensitivity to adversarial noise. The method shows significant robustness improvements against various attacks, particularly on the CIFAR-10 dataset.

The article discusses a new framework for differential privacy auditing in machine learning systems. Traditional methods require altering training datasets, which can be resource-intensive. The proposed observational auditing framework utilizes the randomness of data distributions to evaluate privacy without modifying the original dataset. This approach extends privacy auditing to protected attributes, including labels, addressing significant gaps in existing techniques. Experiments conducted on Criteo and CIFAR-10 datasets validate its effectiveness.

MI-to-Mid Distilled Compression (M2M-DC) is a novel compression framework that combines information-guided block pruning with progressive inner slicing and staged knowledge distillation. The method ranks residual blocks based on a mutual information signal, removing the least informative units. It alternates short knowledge distillation phases with channel slicing to maintain computational efficiency while preserving model accuracy. The approach has demonstrated promising results on CIFAR-100, achieving high accuracy with significantly reduced parameters.