Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation

The introduction of the Fine-grained Cross-modal Causal Tracing (FCCT) framework marks a significant advancement in the mechanistic interpretability of Large Vision-Language Models (LVLMs). Current analyses have been insufficient, failing to comprehensively examine the interactions between visual and textual tokens across various model components and layers. The FCCT framework systematically quantifies causal effects on visual object perception, revealing that multi-head self-attention (MHSA) in the middle layers is critical for aggregating cross-modal information. Additionally, feed-forward networks (FFNs) demonstrate a hierarchical progression in managing visual object representations. This research is pivotal as it not only enhances our understanding of LVLMs but also aids in developing strategies for hallucination mitigation, thereby improving the reliability of AI outputs in practical applications.