DepthVision: Enabling Robust Vision-Language Models with GAN-Based LiDAR-to-RGB Synthesis for Autonomous Driving

Recent advancements in autonomous driving have investigated the application of vision-language models (VLMs) in visual question answering (VQA) frameworks for driving decision-making. However, these methods often rely on handcrafted prompts and exhibit inconsistent performance, which hampers their effectiveness in real-world scenarios. This study assesses state-of-the-art open-source VLMs on high-level decision-making tasks using ego-view visual inputs, revealing significant limitations in their ability to provide reliable, context-aware decisions.

The article presents a new framework for GNSS-free collaborative perception using LiDAR localization, addressing the challenges faced in GNSS-denied environments. Traditional localization methods often struggle in these settings, hindering effective collaboration among multi-agent systems. The proposed solution includes a lightweight Pose Generator with Confidence (PGC) for estimating poses and confidence, alongside the Pose-Aware Spatio-Temporal Alignment Transformer (PASTAT) for spatial alignment. A new simulation dataset, V2VLoc, is introduced, which supports LiDAR localization and collabor…

Recent advancements in vision-language models (VLMs) have utilized large language models (LLMs) to achieve performance comparable to proprietary systems like GPT-4V. However, deploying these models on resource-constrained devices poses challenges due to high computational requirements. To address this, a new framework called Generation after Recalibration (GenRecal) has been introduced, which distills knowledge from large VLMs into smaller, more efficient models by aligning feature representations across diverse architectures.

Nighttime depth estimation using camera systems poses significant challenges, particularly for autonomous driving where accurate depth perception is crucial. Traditional models trained on daytime data often struggle without expensive LiDAR systems. This study introduces Light Enhanced Depth (LED), a novel approach that utilizes high-definition headlights to improve depth estimation in low-light conditions. LED demonstrates substantial performance improvements across various depth-estimation architectures on both synthetic and real datasets.

The article presents a new framework called GMAT, which enhances Multiple Instance Learning (MIL) for whole slide image (WSI) classification. By integrating vision-language models (VLMs), GMAT aims to improve the generation of clinical descriptions that are more expressive and medically specific. This addresses limitations in existing methods that rely on large language models (LLMs) for generating descriptions, which often lack domain grounding and detailed medical specificity, thus improving alignment with visual features.

This paper investigates the role of attention heads in CLIP's image encoder. It finds that certain heads across layers can negatively impact representations. To address this, the authors propose the Attention Ablation Technique (AAT), which suppresses selected heads by manipulating their attention weights. AAT allows for the identification and ablation of harmful heads with minimal overhead, leading to improved downstream performance, including an 11.1% boost in recall on cross-modal retrieval benchmarks.

TAPIP3D is a new method for long-term 3D point tracking in monocular RGB and RGB-D videos. It utilizes camera-stabilized spatio-temporal feature clouds to convert 2D video features into a 3D space, effectively negating camera movement. The approach refines multi-frame motion estimates to enhance point tracking over extended periods. A novel 3D Neighborhood-to-Neighborhood attention mechanism is introduced to manage irregular 3D point distributions, significantly improving performance compared to existing methods and even surpassing state-of-the-art 2D pixel trackers.