arXiv:2601.11632v3 Announce Type: replace 
Abstract: Multi-modal Large Language Models (MLLMs) for Visual Question Answering (VQA) often suffer from dual limitations: knowledge hallucination and insufficient fine-grained visual perception. Crucially, we identify that commonsense graphs and scene graphs provide precisely complementary solutions to these respective deficiencies by providing rich external knowledge and capturing fine-grained visual details. However, prior works typically treat them in isolation, overlooking their synergistic potential. To bridge this gap, we propose KG-ViP, a unified framework that empowers MLLMs by fusing scene graphs and commonsense graphs. The core of the KG-ViP framework is a novel retrieval-and-fusion pipeline that utilizes the query as a semantic bridge to progressively integrate both graphs, synthesizing a unified structured context that facilitates reliable multi-modal reasoning. Extensive experiments on FVQA 2.0+ and MVQA benchmarks demonstrate that KG-ViP significantly outperforms existing VQA methods.

يقدم KG-ViP إطارًا موحدًا يدمج الرسوم البيانية للمعرفة العامة ورسوم المشهد لتعزيز قدرات نماذج اللغة متعددة الوسائط (MLLM) في الإجابة على الأسئلة المرئية (VQA). تتناول هذه المقاربة التحديات المتمثلة في هلاوس المعرفة والقدرة البصرية المحدودة من خلال إنشاء سياق منظم لتحسين التفكير.

KG-ViP presenta un marco unificado que integra gráficos de sentido común y gráficos de escena para mejorar las capacidades de los Modelos de Lenguaje Multimodal (MLLM) en la Respuesta a Preguntas Visuales (VQA). Este enfoque aborda los desafíos de la alucinación de conocimiento y la percepción visual limitada al crear un contexto estructurado para un razonamiento mejorado.

KG-ViP introduit un cadre unifié qui intègre des graphes de bon sens et des graphes de scène pour améliorer les capacités des modèles de langage multimodal (MLLM) dans le domaine de la réponse à des questions visuelles (VQA). Cette approche répond aux défis de l'hallucination de connaissances et de la perception visuelle limitée en créant un contexte structuré pour un raisonnement amélioré.

KG-ViP introduces a unified framework that integrates commonsense graphs and scene graphs to enhance the capabilities of Multi-modal Large Language Models (MLLMs) in Visual Question Answering (VQA). This approach addresses the dual challenges of knowledge hallucination and limited visual perception by creating a structured context for improved reasoning.