arXiv:2511.05923v3 Announce Type: replace 
Abstract: Despite the remarkable advancements of Large Vision-Language Models (LVLMs), the mechanistic interpretability remains underexplored. Existing analyses are insufficiently comprehensive and lack examination covering visual and textual tokens, model components, and the full range of layers. This limitation restricts actionable insights to improve the faithfulness of model output and the development of downstream tasks, such as hallucination mitigation. To address this limitation, we introduce Fine-grained Cross-modal Causal Tracing (FCCT) framework, which systematically quantifies the causal effects on visual object perception. FCCT conducts fine-grained analysis covering the full range of visual and textual tokens, three core model components including multi-head self-attention (MHSA), feed-forward networks (FFNs), and hidden states, across all decoder layers. Our analysis is the first to demonstrate that MHSAs of the last token in middle layers play a critical role in aggregating cross-modal information, while FFNs exhibit a three-stage hierarchical progression for the storage and transfer of visual object representations. Building on these insights, we propose Intermediate Representation Injection (IRI), a training-free inference-time technique that reinforces visual object information flow by precisely intervening on cross-modal representations at specific components and layers, thereby enhancing perception and mitigating hallucination. Consistent improvements across five widely used benchmarks and LVLMs demonstrate IRI achieves state-of-the-art performance, while preserving inference speed and other foundational performance.

تقدم الدراسة إطارًا لتتبع الأسباب بين النماذج اللغوية البصرية الكبيرة (LVLMs) يهدف إلى تحسين قابلية التفسير الميكانيكي. تتناول الدراسة القيود الموجودة في التحليلات الحالية من خلال قياس التأثيرات السببية على إدراك الكائنات البصرية عبر مكونات النموذج المختلفة وطبقاته. من المتوقع أن يحسن هذا النهج موثوقية مخرجات النموذج ويساعد في التخفيف من الهلوسة في LVLMs.

El estudio presenta un marco de trazado causal intermodal detallado (FCCT) destinado a mejorar la interpretabilidad mecanicista de los Modelos de Lenguaje-Visión a Gran Escala (LVLMs). Aborda las limitaciones de los análisis existentes al cuantificar sistemáticamente los efectos causales sobre la percepción de objetos visuales a través de varios componentes y capas del modelo. Se espera que este enfoque mejore la fiabilidad de las salidas del modelo y ayude a mitigar las alucinaciones en los LVLMs.

L'étude présente un cadre de traçage causal intermodal détaillé (FCCT) visant à améliorer l'interprétabilité mécaniste des grands modèles de vision-langage (LVLMs). Elle aborde les limites des analyses existantes en quantifiant systématiquement les effets causaux sur la perception des objets visuels à travers divers composants et couches du modèle. Cette approche devrait améliorer la fiabilité des sorties du modèle et aider à atténuer les hallucinations dans les LVLMs.

The study presents a Fine-grained Cross-modal Causal Tracing (FCCT) framework aimed at enhancing the mechanistic interpretability of Large Vision-Language Models (LVLMs). It addresses the limitations of existing analyses by systematically quantifying causal effects on visual object perception across various model components and layers. This approach is expected to improve the reliability of model outputs and assist in mitigating hallucinations in LVLMs.

Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation

One More Thing in AI – Your Shortcut to AI Mastery

Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

Humanize AI

The Visualizer

LangWatch

OpenL Translator

Ready to build your own newsroom?