PAS : Prelim Attention Score for Detecting Object Hallucinations in Large Vision--Language Models

Large vision-language models (LVLMs) are increasingly recognized for their capabilities, but they face challenges due to object hallucinations. This study reveals that LVLMs often disregard the actual image and instead depend on previously generated output tokens to predict new objects. The research quantifies this behavior by analyzing the mutual information between the image and the predicted object, highlighting a strong correlation between weak image dependence and hallucination. The authors introduce the Prelim Attention Score (PAS), a novel, lightweight metric that can detect object hallucinations effectively without additional training.