MambaEye: A Size-Agnostic Visual Encoder with Causal Sequential Processing

Recent advancements in the optimization of Vision Transformers (ViTs) have been achieved through the integration of Decorrelated Backpropagation (DBP) into Masked Autoencoder (MAE) pre-training, resulting in a 21.1% reduction in wall-clock time and a 21.4% decrease in carbon emissions during training on datasets like ImageNet-1K and ADE20K.

A new study proposes a branch-based structural reparameterization technique for Vision Transformers, aiming to reduce the number of stacked transformer layers while maintaining their representational capacity. This method operates during the training phase, allowing for the consolidation of parallel branches into streamlined models for efficient inference deployment.

Recent research highlights the need for a deeper understanding of latent space in Latent Diffusion Models (LDMs), revealing that these models exhibit uneven memorization across latent codes and that different dimensions within a single latent code contribute variably to memorization. This study introduces a method to rank these dimensions based on their impact on the decoder pullback metric.

A new method called Channel-Aware Typical Set Refinement (TSRE) has been proposed for Out-of-Distribution (OOD) detection, addressing the limitations of existing activation-based methods that often neglect channel characteristics, leading to inaccurate typical set estimations. This method enhances the separation between in-distribution and OOD data, improving model reliability in open-world environments.

The introduction of Velocity Contrastive Regularization (VeCoR) enhances Flow Matching (FM) by implementing a balanced attract-repel scheme, which guides the learned velocity field towards stable directions while avoiding off-manifold errors. This development aims to improve stability and generalization in generative modeling, particularly in lightweight configurations.

The CascadedViT (CViT) architecture introduces a lightweight and compute-efficient Vision Transformer, featuring the innovative Cascaded-Chunk Feed Forward Network (CCFFN), which enhances parameter and FLOP efficiency while maintaining accuracy. Experiments on ImageNet-1K indicate that the CViT-XL model achieves 75.5% Top-1 accuracy, reducing FLOPs by 15% and energy consumption by 3.3% compared to EfficientViT-M5.