arXiv:2508.12711v3 Announce Type: replace 
Abstract: The proliferation of multimodal misinformation poses growing threats to public discourse and societal trust. While Large Vision-Language Models (LVLMs) have enabled recent progress in multimodal misinformation detection (MMD), the rise of generative AI (GenAI) tools introduces a new challenge: GenAI-driven news diversity, characterized by highly varied and complex content. We show that this diversity induces multi-level drift, comprising (1) model-level misperception drift, where stylistic variations disrupt a model's internal reasoning, and (2) evidence-level drift, where expression diversity degrades the quality or relevance of retrieved external evidence. These drifts significantly degrade the robustness of current LVLM-based MMD systems. To systematically study this problem, we introduce DriftBench, a large-scale benchmark comprising 16,000 news instances across six categories of diversification. We design three evaluation tasks: (1) robustness of truth verification under multi-level drift; (2) susceptibility to adversarial evidence contamination generated by GenAI; and (3) analysis of reasoning consistency across diverse inputs. Experiments with six state-of-the-art LVLM-based detectors show substantial performance drops (average F1 -14.8%) and increasingly unstable reasoning traces, with even more severe failures under adversarial evidence injection. Our findings uncover fundamental vulnerabilities in existing MMD systems and suggest an urgent need for more resilient approaches in the GenAI era.

أدى ظهور أدوات الذكاء الاصطناعي التوليدية (GenAI) إلى تحديات كبيرة لأنظمة الكشف عن المعلومات المضللة المعتمدة على نماذج اللغة والرؤية الكبيرة (LVLM). تؤدي هذه التنوعات في محتوى الأخبار إلى انحرافات متعددة المستويات، تؤثر على كل من تفكير النموذج وملاءمة الأدلة الخارجية. تم تقديم معيار جديد، DriftBench، لدراسة هذه المشكلات بشكل منهجي، ويتضمن 16,000 حالة أخبار عبر ست فئات من التنوع.

El auge de las herramientas de inteligencia artificial generativa (GenAI) ha introducido desafíos significativos para los sistemas de detección de desinformación basados en Modelos de Lenguaje y Visión de Gran Escala (LVLM). Esta diversidad en el contenido de noticias conduce a un desplazamiento a múltiples niveles, afectando tanto el razonamiento del modelo como la relevancia de la evidencia externa. Se ha introducido un nuevo benchmark, DriftBench, para estudiar sistemáticamente estos problemas, que comprende 16,000 instancias de noticias en seis categorías de diversificación.

L'essor des outils d'intelligence artificielle générative (GenAI) a introduit des défis significatifs pour les systèmes de détection de désinformation basés sur des modèles de vision-langage de grande taille (LVLM). Cette diversité dans le contenu des nouvelles entraîne un dérive à plusieurs niveaux, affectant à la fois le raisonnement des modèles et la pertinence des preuves externes. Un nouveau benchmark, DriftBench, a été introduit pour étudier systématiquement ces problèmes, comprenant 16 000 instances de nouvelles à travers six catégories de diversification.

The rise of generative AI (GenAI) tools has introduced significant challenges to misinformation detection systems based on Large Vision-Language Models (LVLMs). This diversity in news content leads to multi-level drift, affecting both model reasoning and the relevance of external evidence. A new benchmark, DriftBench, has been introduced to systematically study these issues, comprising 16,000 news instances across six diversification categories.

Drifting Away from Truth: GenAI-Driven News Diversity Challenges LVLM-Based Misinformation Detection

arXiv:2511.17254v1 Announce Type: new 
Abstract: Despite their impressive performance across a wide range of tasks, Large Vision-Language Models (LVLMs) remain prone to hallucination. In this study, we propose a comprehensive intervention framework aligned with the transformer's causal architecture in LVLMs, integrating the effects of different intervention paths on hallucination. We find that hallucinations in LVLMs do not arise from a single causal path, but rather from the interplay among image-to-input-text, image-to-output-text, and text-to-text pathways. For the first time, we also find that LVLMs rely on different pathways depending on the question-answer alignment format. Building on these insights, we propose simple yet effective methods to identify and intervene on critical hallucination heads within each pathway, tailored to discriminative and generative formats. Experiments across multiple benchmarks demonstrate that our approach consistently reduces hallucinations across diverse alignment types.

تقدم دراسة جديدة إطار العمل Intervene-All-Paths، الذي يهدف إلى التخفيف من الهلوسات في نماذج اللغة الكبيرة للرؤية (LVLMs) من خلال معالجة تفاعل مسارات سببية متنوعة. تبرز هذه البحث أن الهلوسات تنشأ من مصادر متعددة، بما في ذلك التفاعلات من الصورة إلى نص الإدخال ومن النص إلى النص، وتقترح تدخلات مستهدفة لأشكال محاذاة الأسئلة والأجوبة المختلفة.

Un nuevo estudio presenta el marco Intervene-All-Paths, destinado a mitigar las alucinaciones en los Modelos de Lenguaje de Visión Grande (LVLMs) al abordar la interacción de diversas rutas causales. Esta investigación destaca que las alucinaciones provienen de múltiples fuentes, incluidas las interacciones de imagen a texto de entrada y de texto a texto, y propone intervenciones específicas para diferentes formatos de alineación de preguntas y respuestas.

Une nouvelle étude présente le cadre Intervene-All-Paths, visant à atténuer les hallucinations dans les grands modèles de vision-langage (LVLMs) en abordant l'interaction de divers chemins causaux. Cette recherche met en évidence que les hallucinations proviennent de multiples sources, y compris les interactions image-texte d'entrée et texte-texte, et propose des interventions ciblées pour différents formats d'alignement question-réponse.

A new study introduces the Intervene-All-Paths framework, aimed at mitigating hallucinations in Large Vision-Language Models (LVLMs) by addressing the interplay of various causal pathways. This research highlights that hallucinations stem from multiple sources, including image-to-input-text and text-to-text interactions, and proposes targeted interventions for different question-answer alignment formats.

Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats

arXiv:2511.11005v2 Announce Type: replace 
Abstract: While recent Large Vision-Language Models (LVLMs) exhibit strong multimodal reasoning abilities, they often produce ungrounded or hallucinated responses because they rely too heavily on linguistic priors instead of visual evidence. This limitation highlights the absence of a quantitative measure of how much these models actually use visual information during reasoning. We propose Draft and Refine (DnR), an agent framework driven by a question-conditioned utilization metric. The metric quantifies the model's reliance on visual evidence by first constructing a query-conditioned relevance map to localize question-specific cues and then measuring dependence through relevance-guided probabilistic masking. Guided by this metric, the DnR agent refines its initial draft using targeted feedback from external visual experts. Each expert's output (such as boxes or masks) is rendered as visual cues on the image, and the model is re-queried to select the response that yields the largest improvement in utilization. This process strengthens visual grounding without retraining or architectural changes. Experiments across VQA and captioning benchmarks show consistent accuracy gains and reduced hallucination, demonstrating that measuring visual utilization provides a principled path toward more interpretable and evidence-driven multimodal agent systems. Code is available at https://github.com/EavnJeong/Draft-and-Refine-with-Visual-Experts.

أدت التطورات الأخيرة في نماذج اللغة والرؤية الكبيرة (LVLM) إلى تقديم إطار العمل Draft and Refine (DnR)، الذي يعزز قدرات التفكير في النماذج من خلال قياس اعتمادها على الأدلة البصرية عبر مقياس استخدام مشروط بالأسئلة. تهدف هذه الطريقة إلى تقليل الاستجابات غير المستندة أو الوهمية من خلال تحسين المسودات الأولية باستخدام تعليقات مستهدفة من خبراء بصريين.

Los recientes avances en los Modelos de Lenguaje-Visión de Gran Escala (LVLM) han llevado a la introducción del marco Draft and Refine (DnR), que mejora las capacidades de razonamiento de los modelos al cuantificar su dependencia de la evidencia visual a través de una métrica de utilización condicionada por preguntas. Este enfoque busca reducir las respuestas infundadas o alucinadas refinando los borradores iniciales con comentarios específicos de expertos visuales.

Des avancées récentes dans les modèles de langage-vision de grande taille (LVLM) ont conduit à l'introduction du cadre Draft and Refine (DnR), qui améliore les capacités de raisonnement des modèles en quantifiant leur dépendance à l'égard des preuves visuelles grâce à une métrique d'utilisation conditionnée par des questions. Cette approche vise à réduire les réponses non fondées ou hallucinées en affinant les brouillons initiaux avec des retours ciblés d'experts visuels.

Recent advancements in Large Vision-Language Models (LVLMs) have led to the introduction of the Draft and Refine (DnR) framework, which enhances the models' reasoning capabilities by quantifying their reliance on visual evidence through a question-conditioned utilization metric. This approach aims to reduce ungrounded or hallucinated responses by refining initial drafts with targeted feedback from visual experts.

Drifting Away from Truth: GenAI-Driven News Diversity Challenges LVLM-Based Misinformation Detection

Was this article worth reading? Share it

ZeroGPT.org

Meteoria

Dyad