Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Driving

Backdoor attacks represent a significant risk to deep learning, particularly in critical 3D applications like autonomous driving and robotics. Current methods primarily focus on static one-to-one attacks, leaving the more versatile one-to-N backdoor threat largely unaddressed. The introduction of STONE (Spherical Trigger One-to-N Backdoor Enabling) marks a pivotal advancement, offering a configurable spherical trigger that can manipulate multiple output labels while maintaining high accuracy in clean data.

MMEdge is a proposed framework designed to enhance real-time multimodal inference on resource-constrained edge devices, crucial for applications like autonomous driving and mobile health. It addresses the challenges of sensing dynamics and inter-modality dependencies by breaking down the inference process into fine-grained sensing and encoding units. This allows for incremental computation as data is received, while a lightweight temporal aggregation module ensures accuracy by capturing rich temporal dynamics across different units.

Recent advancements in autonomous driving have investigated the application of vision-language models (VLMs) in visual question answering (VQA) frameworks for driving decision-making. However, these methods often rely on handcrafted prompts and exhibit inconsistent performance, which hampers their effectiveness in real-world scenarios. This study assesses state-of-the-art open-source VLMs on high-level decision-making tasks using ego-view visual inputs, revealing significant limitations in their ability to provide reliable, context-aware decisions.

The paper introduces Risk Semantic Distillation (RSD), a novel framework aimed at enhancing end-to-end autonomous driving (AD) systems. While current AD systems perform well in complex scenarios, they struggle with generalization to unseen situations. RSD leverages Vision-Language Models (VLMs) to improve training efficiency and consistency in trajectory planning, addressing challenges posed by hybrid AD systems that utilize multiple planning approaches. This advancement is crucial for the future of autonomous driving technology.

A recent study has introduced a novel physical adversarial attack targeting stereo matching models used in autonomous driving. Unlike traditional attacks that utilize 2D patches, this method employs a 3D physical adversarial example (PAE) with global camouflage texture, enhancing visual consistency across various viewpoints of stereo cameras. The research also presents a new 3D stereo matching rendering module to align the PAE with real-world positions, addressing the disparity effects inherent in binocular vision.

The article discusses the growing interest in world models, particularly in the context of advancements in multimodal large language models like GPT-4 and video generation models such as Sora. It provides a comprehensive review of the literature on world models, which serve to either understand the current state of the world or predict future dynamics. The review categorizes world models based on their functions: constructing internal representations and predicting future states, with applications in generative games, autonomous driving, robotics, and social simulacra.

The paper titled 'Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning' addresses the challenges of high-variance return estimates in reinforcement learning algorithms. It highlights that well-designed behavior policies can collect off-policy data, leading to lower variance return estimates. This finding suggests that on-policy data collection is not optimal for variance, and the authors extend this insight to online reinforcement learning, where policy evaluation and improvement occur simultaneously.

The study investigates the ability of transformer models to predict long steps in the Collatz sequence, a complex arithmetic function that maps odd integers to their successors. The accuracy of the models varies significantly depending on the base used for encoding, achieving up to 99.7% accuracy for bases 24 and 32, while dropping to 37% and 25% for bases 11 and 3. Despite these variations, all models exhibit a common learning pattern, accurately predicting inputs with similar residuals modulo 2^p.