LTD: Low Temperature Distillation for Gradient Masking-free Adversarial Training

A new framework called EntPruner has been introduced to address parameter redundancy in large-scale vision generative models, specifically diffusion and flow models. This framework employs an entropy-guided automatic progressive pruning strategy, which assesses the importance of model blocks based on Conditional Entropy Deviation (CED) to optimize performance across various downstream tasks.

The introduction of Filter Like You Test (FLYT) presents a novel algorithm for curating large-scale vision-language datasets, enhancing the selection of pretraining examples by learning the usefulness of each data point through gradient signals from downstream tasks. This is complemented by Mixing-FLYT (M-FLYT) and Soft Cap Sampling (SCS), which improve dataset filtering and accuracy.

A novel framework called Dynamic Epsilon Scheduling (DES) has been proposed to enhance adversarial training for deep neural networks. This approach adapts the adversarial perturbation budget based on instance-specific characteristics, integrating factors such as distance to decision boundaries, prediction confidence, and model uncertainty. This advancement addresses the limitations of fixed perturbation budgets in existing methods.

A comprehensive study has been conducted comparing three generative modeling paradigms: Denoising Diffusion Probabilistic Models (DDPM), Conditional Flow Matching (CFM), and MeanFlow, focusing on their application in image inpainting. The study highlights that CFM significantly outperforms DDPM in terms of efficiency and quality, achieving a notable FID score of 24.15 with only 50 steps, while MeanFlow allows for single-step generation, reducing inference time by 50 times.

A recent study on Vision Transformers (ViTs) reveals a non-monotonic scaling behavior, where deeper models like ViT-L may underperform compared to shallower variants such as ViT-S and ViT-B. This research identifies a three-phase pattern—Cliff-Plateau-Climb—indicating how representation quality evolves with depth, particularly noting the diminishing role of the [CLS] token in favor of patch tokens for better performance.

The Stability-Guided Online Influence Framework (SG-OIF) has been introduced to enhance the reliability of vision data in deep learning models, addressing challenges such as the computational expense of influence function implementations and the instability of training dynamics. This framework aims to provide real-time control over algorithmic stability, facilitating more accurate identification of critical training examples.

The introduction of DP-MicroAdam marks a significant advancement in the realm of adaptive optimizers for differentially private training, demonstrating superior performance and convergence rates compared to traditional methods like DP-SGD. This new algorithm is designed to be memory-efficient and sparsity-aware, addressing the challenges of extensive compute and hyperparameter tuning typically associated with differential privacy.

A recent study titled 'ModHiFi: Identifying High Fidelity predictive components for Model Modification' explores methods to modify open weight models without access to training data or loss functions. The research focuses on identifying critical components that influence predictive performance using only distributional access, such as synthetic data.