arXiv:2511.10671v1 Announce Type: new 
Abstract: Visual hallucination, where Multimodal Large Language Models fabricate details inconsistent with image content, critically undermines their reliability. Existing fine-tuning methods offer limited improvement, failing to deeply intervene in factual reasoning. This paper introduces Grounded Visual Factualization (GVF) Finetuning, a novel approach to systematically enhance MLLM visual factual consistency. GVF integrates explicit factual signals via three core mechanisms: Factual Anchor Data Augmentation, enriching training data with structured factual anchors and counter-factual prompts; Fact-Aware Instruction Tuning, embedding these cues into explicit instructions; and a Factual Consistency Loss function, specifically penalizing factual inaccuracies. Evaluated on LLaVA-1.5-13B, GVF Finetuning significantly outperforms standard fine-tuning on the VHTest benchmark for both Open-Ended Question (OEQ) and Yes/No Question (YNQ) formats. Crucially, GVF maintains or even slightly improves performance on general multimodal benchmarks like MME and POPE, demonstrating effective mitigation of visual hallucinations without compromising general understanding and reasoning abilities.

يتناول المقال بعنوان 'Grounded Visual Factualization: Factual Anchor-Based Finetuning for Enhancing MLLM Factual Consistency' مشكلة الهلوسة البصرية في نماذج اللغة متعددة الوسائط (MLLM)، حيث تقوم هذه النماذج بإنشاء تفاصيل غير متوافقة مع محتوى الصور. أظهرت طرق الضبط الحالية تحسينات محدودة في التفكير الواقعي. يقترح المؤلفون نهجًا جديدًا يسمى Grounded Visual Factualization (GVF) Finetuning، والذي يعزز الاتساق الواقعي البصري من خلال ثلاثة آليات: زيادة بيانات مرجعية واقعية، وضبط التعليمات الواعية بالحقائق، ودالة خسارة الاتساق الواقعي. تظهر التقييمات على LLaVA-1.5-13B أن GVF يتفوق بشكل كبير على طرق ا…

El artículo titulado 'Grounded Visual Factualization: Factual Anchor-Based Finetuning for Enhancing MLLM Factual Consistency' aborda el problema de la alucinación visual en los Modelos de Lenguaje Multimodal (MLLM), donde estos modelos generan detalles que son inconsistentes con el contenido de las imágenes. Los métodos de ajuste actuales han mostrado mejoras limitadas en el razonamiento factual. Los autores proponen un nuevo enfoque llamado Grounded Visual Factualization (GVF) Finetuning, que mejora la consistencia factual visual a través de tres mecanismos: Aumento de Datos de Anclaje Factua…

L'article intitulé 'Grounded Visual Factualization: Factual Anchor-Based Finetuning for Enhancing MLLM Factual Consistency' traite du problème de l'hallucination visuelle dans les modèles de langage multimodaux (MLLM), où ces modèles génèrent des détails qui ne correspondent pas aux images accompagnantes. Les méthodes de réglage actuelles ont montré un succès limité dans l'amélioration du raisonnement factuel. Les auteurs proposent une nouvelle approche appelée Grounded Visual Factualization (GVF) Finetuning, qui améliore la cohérence factuelle visuelle grâce à trois mécanismes : l'augmentatio…

The paper titled 'Grounded Visual Factualization: Factual Anchor-Based Finetuning for Enhancing MLLM Factual Consistency' addresses the issue of visual hallucination in Multimodal Large Language Models (MLLMs), where these models generate details that are inconsistent with the accompanying images. Current fine-tuning methods have shown limited success in improving factual reasoning. The authors propose a new approach called Grounded Visual Factualization (GVF) Finetuning, which enhances visual factual consistency through three mechanisms: Factual Anchor Data Augmentation, Fact-Aware Instructio…

Grounded Visual Factualization: Factual Anchor-Based Finetuning for Enhancing MLLM Factual Consistency

One More Thing in AI – Your Shortcut to AI Mastery

Grounded Visual Factualization: Factual Anchor-Based Finetuning for Enhancing MLLM Factual Consistency

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

The Visualizer

Langfuse

GPTHumanizer

Meteoria

Ready to build your own newsroom?