Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining

A new framework called EntPruner has been introduced to address parameter redundancy in large-scale vision generative models, specifically diffusion and flow models. This framework employs an entropy-guided automatic progressive pruning strategy, which assesses the importance of model blocks based on Conditional Entropy Deviation (CED) to optimize performance across various downstream tasks.

A recent study on Vision Transformers (ViTs) reveals a non-monotonic scaling behavior, where deeper models like ViT-L may underperform compared to shallower variants such as ViT-S and ViT-B. This research identifies a three-phase pattern—Cliff-Plateau-Climb—indicating how representation quality evolves with depth, particularly noting the diminishing role of the [CLS] token in favor of patch tokens for better performance.

A novel approach called Low-Temperature Distillation (LTD) has been introduced to enhance adversarial training in neural networks, addressing the vulnerabilities associated with one-hot label representations in image classification. LTD utilizes a lower temperature in the teacher model while keeping the student model's temperature fixed, refining label representations and improving model robustness against adversarial attacks.

The introduction of DP-MicroAdam marks a significant advancement in the realm of adaptive optimizers for differentially private training, demonstrating superior performance and convergence rates compared to traditional methods like DP-SGD. This new algorithm is designed to be memory-efficient and sparsity-aware, addressing the challenges of extensive compute and hyperparameter tuning typically associated with differential privacy.

A recent study titled 'ModHiFi: Identifying High Fidelity predictive components for Model Modification' explores methods to modify open weight models without access to training data or loss functions. The research focuses on identifying critical components that influence predictive performance using only distributional access, such as synthetic data.

The introduction of Multi-Granularity Differentiable Architecture Search (MG-DARTS) marks a significant advancement in neural architecture search (NAS), focusing on optimizing both model effectiveness and efficiency. This framework addresses limitations in existing differentiable architecture search methods by incorporating finer-grained structures, enhancing the balance between model performance and size.

The newly proposed DeCo framework introduces a frequency-decoupled pixel diffusion method for end-to-end image generation, addressing the inefficiencies of existing models that combine high and low-frequency signal modeling within a single diffusion transformer. This innovation allows for improved training and inference speeds by separating the generation processes of high-frequency details and low-frequency semantics.

A new study introduces Temporal-adaptive Weight Quantization (TaWQ) for Spiking Neural Networks (SNNs), which aims to reduce energy consumption while maintaining accuracy. This method leverages temporal dynamics to allocate ultra-low-bit weights, demonstrating minimal quantization loss of 0.22% on ImageNet and high energy efficiency in extensive experiments.