Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs

Virtual keyboards in augmented reality (AR) often frustrate users due to their slow response and high error rates. Users experience discomfort, commonly referred to as 'gorilla arm,' from raising their arms to type on these virtual surfaces.

The article presents Uni-Hand, a universal hand motion forecasting framework designed for egocentric views. This framework addresses challenges in hand trajectory prediction methods, such as insufficient prediction targets and entangled hand-head motion. By utilizing multi-modal inputs and incorporating vision-language fusion, it aims to enhance applications in augmented reality and human-robot interaction. The framework forecasts hand waypoints in both 2D and 3D spaces, improving the accuracy of motion predictions.

Wave-Former is a new method for high-accuracy 3D shape reconstruction of completely occluded everyday objects. Utilizing millimeter-wave (mmWave) wireless signals, it can penetrate common obstructions and reflect off hidden items. Unlike previous methods that faced limitations in coverage and noise, Wave-Former employs a physics-aware shape completion model to infer full 3D geometry. Its innovative three-stage pipeline connects raw wireless signals with advancements in vision-based shape completion, enhancing applications in robotics, augmented reality, and logistics.

The Geometry-guided Multi-View Diffusion Model (GeoMVD) has been proposed to enhance multi-view image generation, addressing challenges in maintaining cross-view consistency and producing high-resolution outputs. This model utilizes geometric information extraction techniques, including depth maps and normal maps, to create images that are structurally consistent and rich in detail. The advancements in this model hold significant implications for applications in computer vision, such as 3D reconstruction and augmented reality.

Large vision-language models (LVLMs) are increasingly recognized for their capabilities, but they face challenges due to object hallucinations. This study reveals that LVLMs often disregard the actual image and instead depend on previously generated output tokens to predict new objects. The research quantifies this behavior by analyzing the mutual information between the image and the predicted object, highlighting a strong correlation between weak image dependence and hallucination. The authors introduce the Prelim Attention Score (PAS), a novel, lightweight metric that can detect object hallucinations effectively without additional training.

The paper titled 'Dynamic Gaussian Scene Reconstruction from Unsynchronized Videos' presents a novel approach to multi-view video reconstruction, crucial for applications in computer vision, film production, virtual reality, and motion analysis. The authors address the common issue of temporal misalignment in unsynchronized video streams, which can degrade reconstruction quality. They propose a temporal alignment strategy that utilizes a coarse-to-fine alignment module to estimate and compensate for time shifts between cameras, enhancing the overall reconstruction process.

An AI-assisted Augmented Reality (AR) assembly workflow has been developed, utilizing deep learning-based object recognition to identify assembly components and provide step-by-step instructions. The system displays bounding boxes around components in real-time, indicating their placement, thus eliminating the need for manual searching or sorting. A case study involving the assembly of LEGO sculptures demonstrates the system's feasibility and effectiveness in enhancing the assembly process.

TEyeD is the world's largest unified public dataset of eye images, featuring over 20 million images collected using seven different head-mounted eye trackers, including devices integrated into virtual and augmented reality systems. The dataset encompasses a variety of activities, such as car rides and sports, and includes detailed annotations like 2D and 3D landmarks, semantic segmentation, and gaze vectors. This resource aims to enhance research in computer vision, eye tracking, and gaze estimation.