Application of Graph Based Vision Transformers Architectures for Accurate Temperature Prediction in Fiber Specklegram Sensors

Feature-map knowledge distillation (KD) is effective for convolutional networks but often fails for Vision Transformers (ViTs). A two-view representation analysis reveals that final-layer representations in ViTs are globally low-rank, suggesting that a compact student model should suffice for feature alignment. However, a token-level Spectral Energy Pattern analysis shows that individual tokens distribute energy across many channels, indicating a mismatch in encoding.

This paper introduces a novel approach to self pre-training using topology- and spatiality-aware Masked Autoencoders (MAEs) for 3D medical image segmentation. The proposed method enhances the ability of Vision Transformers (ViTs) to capture geometric shape and spatial information, which are crucial for accurate segmentation. A new topological loss is introduced to preserve geometric shape information, improving the performance of MAEs in medical imaging tasks.

Vision Transformers (ViTs) have become the leading architecture for visual processing tasks, showcasing remarkable scalability with larger training datasets and model sizes. However, recent findings have revealed the presence of artifact tokens in ViTs that conflict with local semantics, negatively impacting performance in tasks requiring precise localization and structural coherence. This paper introduces register tokens to mitigate this issue, proposing Post Hoc Registers (PH-Reg) as an efficient self-distillation method to integrate these tokens into existing ViTs without the need for retra…

The article presents a novel approach to optimizing vision transformer (ViT) models by creating a stratified knowledge-density super-network. This method organizes knowledge hierarchically across weights, allowing for flexible extraction of sub-networks that maintain essential knowledge for various model sizes. The introduction of Weighted PCA for Attention Contraction (WPAC) enhances knowledge compactness while preserving the original network function, addressing the inefficiencies of training multiple ViT models under different resource constraints.

The paper introduces a novel likelihood-guided variational Ising-based regularization framework for Vision Transformers (ViTs), aimed at enhancing model generalization while dynamically pruning redundant parameters. This approach utilizes Bayesian sparsification techniques to impose structured sparsity on model weights, allowing for adaptive architecture search during training. Unlike traditional dropout methods, this framework learns task-adaptive regularization, improving efficiency and interpretability in classification tasks involving structured and high-dimensional data.