CascadedViT: Cascaded Chunk-FeedForward and Cascaded Group Attention Vision Transformer

Feature-map knowledge distillation (KD) is effective for convolutional networks but often fails for Vision Transformers (ViTs). A two-view representation analysis reveals that final-layer representations in ViTs are globally low-rank, suggesting that a compact student model should suffice for feature alignment. However, a token-level Spectral Energy Pattern analysis shows that individual tokens distribute energy across many channels, indicating a mismatch in encoding.

Data-Free Quantization (DFQ) presents a solution for model compression without needing real data, which is beneficial in privacy-sensitive contexts. While DFQ has been effective for unimodal models, its application to Vision-Language Models like CLIP has not been thoroughly investigated. This study introduces D4C, a DFQ framework specifically designed for CLIP, addressing challenges such as semantic content and intra-image diversity in synthesized samples.

The paper titled 'UNSEEN: Enhancing Dataset Pruning from a Generalization Perspective' addresses the computational challenges posed by large datasets in deep learning. It proposes a novel approach to dataset pruning that focuses on generalization rather than fitting, scoring samples based on models not exposed to them during training. This method aims to create a more effective selection process by reducing the concentration of sample scores, ultimately improving the performance of deep learning models.