VLMs Guided Interpretable Decision Making for Autonomous Driving

DepthVision is a multimodal framework designed to enhance Vision-Language Models (VLMs) by utilizing LiDAR data without requiring architectural modifications or retraining. It synthesizes RGB-like images from sparse LiDAR point clouds using a conditional GAN and integrates a Luminance-Aware Modality Adaptation (LAMA) module to dynamically adjust image quality based on ambient lighting. This innovation aims to improve the reliability of autonomous vehicles in challenging visual conditions, such as darkness or motion blur.

MMEdge is a proposed framework designed to enhance real-time multimodal inference on resource-constrained edge devices, crucial for applications like autonomous driving and mobile health. It addresses the challenges of sensing dynamics and inter-modality dependencies by breaking down the inference process into fine-grained sensing and encoding units. This allows for incremental computation as data is received, while a lightweight temporal aggregation module ensures accuracy by capturing rich temporal dynamics across different units.

The paper introduces Risk Semantic Distillation (RSD), a novel framework aimed at enhancing end-to-end autonomous driving (AD) systems. While current AD systems perform well in complex scenarios, they struggle with generalization to unseen situations. RSD leverages Vision-Language Models (VLMs) to improve training efficiency and consistency in trajectory planning, addressing challenges posed by hybrid AD systems that utilize multiple planning approaches. This advancement is crucial for the future of autonomous driving technology.

Recent advancements in vision-language models (VLMs) have utilized large language models (LLMs) to achieve performance comparable to proprietary systems like GPT-4V. However, deploying these models on resource-constrained devices poses challenges due to high computational requirements. To address this, a new framework called Generation after Recalibration (GenRecal) has been introduced, which distills knowledge from large VLMs into smaller, more efficient models by aligning feature representations across diverse architectures.

A recent study has introduced a novel physical adversarial attack targeting stereo matching models used in autonomous driving. Unlike traditional attacks that utilize 2D patches, this method employs a 3D physical adversarial example (PAE) with global camouflage texture, enhancing visual consistency across various viewpoints of stereo cameras. The research also presents a new 3D stereo matching rendering module to align the PAE with real-world positions, addressing the disparity effects inherent in binocular vision.

Backdoor attacks represent a significant risk to deep learning, particularly in critical 3D applications like autonomous driving and robotics. Current methods primarily focus on static one-to-one attacks, leaving the more versatile one-to-N backdoor threat largely unaddressed. The introduction of STONE (Spherical Trigger One-to-N Backdoor Enabling) marks a pivotal advancement, offering a configurable spherical trigger that can manipulate multiple output labels while maintaining high accuracy in clean data.

The article presents a new framework called GMAT, which enhances Multiple Instance Learning (MIL) for whole slide image (WSI) classification. By integrating vision-language models (VLMs), GMAT aims to improve the generation of clinical descriptions that are more expressive and medically specific. This addresses limitations in existing methods that rely on large language models (LLMs) for generating descriptions, which often lack domain grounding and detailed medical specificity, thus improving alignment with visual features.

This paper investigates the role of attention heads in CLIP's image encoder. It finds that certain heads across layers can negatively impact representations. To address this, the authors propose the Attention Ablation Technique (AAT), which suppresses selected heads by manipulating their attention weights. AAT allows for the identification and ablation of harmful heads with minimal overhead, leading to improved downstream performance, including an 11.1% boost in recall on cross-modal retrieval benchmarks.