Benchmarking Visual LLMs Resilience to Unanswerable Questions on Visually Rich Documents

Recent advancements in autonomous driving have investigated the application of vision-language models (VLMs) in visual question answering (VQA) frameworks for driving decision-making. However, these methods often rely on handcrafted prompts and exhibit inconsistent performance, which hampers their effectiveness in real-world scenarios. This study assesses state-of-the-art open-source VLMs on high-level decision-making tasks using ego-view visual inputs, revealing significant limitations in their ability to provide reliable, context-aware decisions.

Geospatial chain of thought (CoT) reasoning is crucial for enhancing Visual Question Answering (VQA) on satellite imagery, especially in climate-related applications like disaster monitoring and urban resilience planning. Current VQA models can interpret remote sensing data but often lack the structured reasoning needed for complex geospatial queries. A new framework integrating CoT reasoning with Direct Preference Optimization (DPO) has been proposed, showing a 34.9% accuracy improvement in handling tasks such as detection and classification.