ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention

The introduction of ViCA (Vision-only Cross-Attention) presents a new architecture for multimodal large language models (MLLMs) that minimizes computational overhead by allowing visual tokens to bypass dense processing layers, interacting with text through selective cross-attention. This approach maintains 98% of baseline accuracy while significantly reducing visual-side computation to just 4%.

This development is crucial as it addresses the inefficiencies of traditional MLLM architectures, potentially leading to faster and more efficient models that can handle multimodal tasks with reduced resource consumption.

The evolution of MLLMs is marked by a trend towards optimizing computational efficiency while enhancing capabilities, as seen in various innovations like Gaze Attention and Vision-OPD, which aim to improve visual understanding and reasoning. These advancements highlight a growing focus on refining the interaction between visual and textual modalities in AI systems.