Thicker and Quicker: A Jumbo Token for Fast Plain Vision Transformers

A new paper introduces an innovative dataset quantization method aimed at reducing storage and communication costs for large-scale datasets on resource-constrained edge devices. This approach focuses on compressing individual samples by minimizing intra-sample redundancy while retaining essential features, marking a shift from traditional inter-sample redundancy methods.

A new study proposes a structured initialization method for Vision Transformers (ViTs), aiming to integrate the strong inductive biases of Convolutional Neural Networks (CNNs) without altering the architecture. This approach is designed to enhance performance on small datasets while maintaining scalability as data increases.

The Iwin Transformer has been introduced as a novel hierarchical vision transformer that operates without position embeddings, utilizing interleaved window attention and depthwise separable convolution to enhance performance across various visual tasks. This architecture allows for direct fine-tuning from low to high resolution, achieving notable results such as 87.4% top-1 accuracy on ImageNet-1K.

A recent study explores the use of predefined vector systems, particularly An root system vectors, to enhance the training of neural networks (NNs) by configuring their latent spaces. This approach allows for training classifiers without classification layers, which is particularly beneficial for datasets with a vast number of classes, such as ImageNet-1K.

Variational Supervised Contrastive Learning (VarCon) has been introduced to enhance supervised contrastive learning by reformulating it as variational inference over latent class variables, addressing limitations in embedding distribution and generalization. This method aims to improve class-aware matching and control intra-class dispersion in the embedding space.