arXiv:2505.21955v2 Announce Type: replace 
Abstract: Large vision-language models (LVLMs) are increasingly deployed in interactive applications such as virtual and augmented reality, where a first-person (egocentric) view captured by head-mounted cameras serves as key input. While this view offers fine-grained cues about user attention and hand-object interactions, its narrow field of view and lack of global context often lead to failures on spatially or contextually demanding queries. To address this, we introduce a framework that augments egocentric inputs with third-person (exocentric) views, providing complementary information such as global scene layout and object visibility to LVLMs. We present E3VQA, the first benchmark for multi-view question answering with 4K high-quality question-answer pairs grounded in synchronized ego-exo image pairs. Additionally, we propose M3CoT, a training-free prompting technique that constructs a unified scene representation by integrating scene graphs from three complementary perspectives. M3CoT enables LVLMs to reason more effectively across views, yielding consistent performance gains (4.84% for GPT-4o and 5.94% for Gemini 2.0 Flash) over a recent CoT baseline. Our extensive evaluation reveals key strengths and limitations of LVLMs in multi-view reasoning and highlights the value of leveraging both egocentric and exocentric inputs. The dataset and source code are available at https://github.com/Leeinsu1/Towards-Comprehensive-Scene-Understanding.

تسلط دراسة حديثة الضوء على دمج وجهات النظر من الشخص الأول والثالث في نماذج اللغة والرؤية الكبيرة (LVLM)، وهو أمر حاسم لتحسين التطبيقات التفاعلية مثل الواقع الافتراضي والمعزز. من خلال دمج الرؤى التفصيلية من وجهات النظر الذاتية مع معلومات سياقية أوسع، يمكن لهذه النماذج تحسين أدائها بشكل كبير في الاستفسارات المكانية المعقدة. لا تعزز هذه التطورات تجربة المستخدم فحسب، بل تفتح أيضًا آفاقًا جديدة لتفاعلات أكثر غمرًا وبديهية في البيئات الرقمية.

Un estudio reciente destaca la integración de vistas de primera y tercera persona en los grandes modelos de lenguaje y visión (LVLM), lo cual es crucial para mejorar aplicaciones interactivas como la realidad virtual y aumentada. Al combinar las percepciones detalladas de las vistas egocéntricas con información contextual más amplia, estos modelos pueden mejorar significativamente su rendimiento en consultas espaciales complejas. Este avance no solo mejora la experiencia del usuario, sino que también abre nuevas oportunidades para interacciones más inmersivas e intuitivas en entornos digitales.

Une étude récente met en avant l'intégration des vues de première et de troisième personne dans les grands modèles de langage et de vision (LVLM), ce qui est essentiel pour améliorer les applications interactives telles que la réalité virtuelle et augmentée. En combinant les informations détaillées des vues égocentriques avec un contexte plus large, ces modèles peuvent considérablement améliorer leurs performances sur des requêtes spatiales complexes. Cette avancée améliore non seulement l'expérience utilisateur, mais ouvre également de nouvelles perspectives pour des interactions plus immersives et intuitives dans les environnements numériques.

A recent study highlights the integration of first and third-person views in large vision-language models (LVLMs), which is crucial for enhancing interactive applications like virtual and augmented reality. By combining the detailed insights from egocentric views with broader contextual information, these models can significantly improve their performance on complex spatial queries. This advancement not only enhances user experience but also opens new avenues for more immersive and intuitive interactions in digital environments.

Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs

One More Thing in AI – Your Shortcut to AI Mastery

Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

Interactive Avatar

Deptho.ai

Attentive AI

The Visualizer

Ready to build your own newsroom?