HybridToken-VLM: Hybrid Token Compression for Vision-Language Models

A new method called Stitch and Tell (SiTe) has been proposed to enhance the spatial understanding of vision-language models, addressing the issue of spatial hallucinations that lead to incorrect descriptions of object positions in images. This method constructs stitched image-text pairs and generates spatially-aware captions without requiring extensive annotations or advanced models.

A new framework named SAVE (Sparse Autoencoder-Driven Visual Information Enhancement) has been proposed to mitigate object hallucination in Multimodal Large Language Models (MLLMs). By steering models along Sparse Autoencoder latent features, SAVE enhances visual understanding and reduces hallucination, achieving significant improvements on benchmarks like CHAIR_S and POPE.

A new technique called Gaussian Quant (GQ) has been introduced to enhance the training of Vector Quantized Variational Autoencoders (VQ-VAE), which are used for compressing images into discrete tokens. This method allows for the conversion of a Gaussian VAE into a VQ-VAE without the need for extensive training, thereby simplifying the process and improving performance.

The Vision Action Transformer (VAT) has been introduced as an innovative architecture that enhances the capabilities of Vision Transformers (ViTs) by utilizing the full feature hierarchy, rather than just the final layer's features. This approach allows VAT to process specialized action tokens alongside visual features across all transformer layers, achieving a remarkable 98.15% success rate on LIBERO benchmarks in simulated manipulation tasks.

DynamicLRP has been introduced as a model-agnostic framework for Layer-wise Relevance Propagation (LRP), allowing for attribution in neural networks without the need for architecture-specific modifications. This innovation operates at the tensor operation level, utilizing a Promise System for deferred activation resolution, thereby enhancing the generality and sustainability of LRP implementations.

A new framework has been proposed to reduce hallucinations in vision-language models (VLMs), which often generate plausible but incorrect claims about image content. This training-free self-correction method allows VLMs to refine their responses through uncertainty-guided visual re-attention, utilizing the Qwen2.5-VL-7B architecture and validated on the POPE and MMHAL BENCH benchmarks.

A new paper introduces data taggants, a technique for dataset ownership verification that utilizes harmless targeted data poisoning to subtly alter datasets. This method aims to address the limitations of existing approaches, such as backdoor watermarking, which can harm model performance and lack guarantees against false positives.