arXiv:2510.11295v2 Announce Type: replace 
Abstract: Large vision-language models (VLMs) achieve strong performance in Visual Question Answering but still rely heavily on supervised fine-tuning (SFT) with massive labeled datasets, which is costly due to human annotations. Crucially, real-world datasets often exhibit human uncertainty (HU) -- variation in human confidence across annotations -- but standard SFT simply optimizes toward the most frequent label, disregarding HU distributions. This leaves two open questions: How does HU affect SFT, and how can HU be effectively leveraged in training? In this work, we first conduct a systematic evaluation of VLMs across varying HU levels. We have two key findings: (i) surprisingly, high-HU samples contribute little or even degrade model performance, and (ii) naively training on the full dataset yields under-calibrated models that fail to capture HU distributions. Motivated by these findings, we introduce HaDola, a human uncertainty-aware data selection and automatic labeling framework. HaDola operates in four stages -- discriminate, self-annotate, error trigger, and training -- to iteratively identify harmful samples, prioritize informative ones, and bootstrap from a small seed set (5\% of data). Our approach substantially reduces reliance on costly HU annotations and makes VLMs more accurate and better calibrated. Extensive experiments on VQAv2 and VizWiz datasets demonstrate that HaDola consistently matches or outperforms state-of-the-art baselines with less training data. Our work highlights the importance of explicitly modeling HU in SFT, suggesting that better utilization of HU is more effective than merely scaling up dataset size.

تسلط دراسة حديثة الضوء على التقدم في مجال الإجابة على الأسئلة المرئية (VQA) من خلال معالجة التحديات التي تطرحها عدم اليقين البشري في تصنيف البيانات. تعتمد الطرق التقليدية بشكل كبير على مجموعات بيانات كبيرة مصنفة، والتي يمكن أن تكون مكلفة وغالبًا ما تتجاهل التباينات في ثقة الإنسان. تقترح هذه الدراسة نهجًا جديدًا لا يحسن فقط كفاءة اختيار البيانات، بل يعزز أيضًا أداء النموذج من خلال دمج هذه الشكوك. هذا مهم لأنه قد يؤدي إلى أنظمة ذكاء اصطناعي أكثر قوة تفهم وتفسر المدخلات البشرية بشكل أفضل، مما يجعل VQA أكثر سهولة وفعالية.

Un estudio reciente destaca los avances en la Respuesta a Preguntas Visuales (VQA) al abordar los desafíos que plantea la incertidumbre humana en el etiquetado de datos. Los métodos tradicionales dependen en gran medida de grandes conjuntos de datos etiquetados, que pueden ser costosos y a menudo pasan por alto las variaciones en la confianza humana. Esta investigación propone un nuevo enfoque que no solo mejora la eficiencia de la selección de datos, sino que también aumenta el rendimiento del modelo al incorporar estas incertidumbres. Esto es significativo, ya que podría llevar a sistemas de IA más robustos que comprendan e interpreten mejor la entrada humana, haciendo que el VQA sea más accesible y efectivo.

Une étude récente met en lumière les avancées dans le domaine de la réponse à des questions visuelles (VQA) en abordant les défis posés par l'incertitude humaine dans l'étiquetage des données. Les méthodes traditionnelles reposent fortement sur de grands ensembles de données étiquetées, qui peuvent être coûteux et négligent souvent les variations de confiance humaine. Cette recherche propose une nouvelle approche qui améliore non seulement l'efficacité de la sélection des données, mais renforce également les performances du modèle en intégrant ces incertitudes. Cela est significatif car cela pourrait conduire à des systèmes d'IA plus robustes qui comprennent et interprètent mieux les entrées humaines, rendant finalement le VQA plus accessible et efficace.

A recent study highlights advancements in Visual Question Answering (VQA) by addressing the challenges posed by human uncertainty in data labeling. Traditional methods rely heavily on large labeled datasets, which can be expensive and often overlook the variations in human confidence. This research proposes a new approach that not only improves the efficiency of data selection but also enhances the model's performance by incorporating these uncertainties. This is significant as it could lead to more robust AI systems that better understand and interpret human input, ultimately making VQA more accessible and effective.

Human Uncertainty-Aware Data Selection and Automatic Labeling in Visual Question Answering

One More Thing in AI – Your Shortcut to AI Mastery

Human Uncertainty-Aware Data Selection and Automatic Labeling in Visual Question Answering

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

Inline Help

The Visualizer

LangWatch

Usercall

Ready to build your own newsroom?