arXiv:2512.06096v1 Announce Type: new 
Abstract: The rapid development of Vision-Language models (VLMs) and Multimodal Language Models (MLLMs) in autonomous driving research has significantly reshaped the landscape by enabling richer scene understanding, context-aware reasoning, and more interpretable decision-making. However, a lot of existing work often relies on either single-view encoders that fail to exploit the spatial structure of multi-camera systems or operate on aggregated multi-view features, which lack a unified spatial representation, making it more challenging to reason about ego-centric directions, object relations, and the wider context. We thus present BeLLA, an end-to-end architecture that connects unified 360{\deg} BEV representations with a large language model for question answering in autonomous driving. We primarily evaluate our work using two benchmarks - NuScenes-QA and DriveLM, where BeLLA consistently outperforms existing approaches on questions that require greater spatial reasoning, such as those involving relative object positioning and behavioral understanding of nearby objects, achieving up to +9.3% absolute improvement in certain tasks. In other categories, BeLLA performs competitively, demonstrating the capability of handling a diverse range of questions.

تم تقديم BeLLA، وهي بنية تحتية شاملة للقيادة الذاتية، والتي تدمج تمثيلات موحدة بزاوية 360 درجة مع نموذج لغة كبير لتحسين الإجابة على الأسئلة. يعالج هذا التطور القيود الموجودة في نماذج اللغة البصرية ونماذج اللغة متعددة الوسائط من خلال تحسين فهم المشهد وعمليات اتخاذ القرار.

Se ha presentado BeLLA, una arquitectura de extremo a extremo para la conducción autónoma, que integra representaciones unificadas de 360° en vista cenital con un gran modelo de lenguaje para mejorar la respuesta a preguntas. Este desarrollo aborda las limitaciones de los modelos de lenguaje visual y los modelos de lenguaje multimodal existentes al mejorar la comprensión de escenas y los procesos de toma de decisiones.

BeLLA, une architecture de bout en bout pour la conduite autonome, a été présentée, intégrant des représentations unifiées à 360° en vue de dessus avec un grand modèle de langage pour améliorer le questionnement. Ce développement répond aux limitations des modèles de langage visuel et des modèles de langage multimodal existants en améliorant la compréhension des scènes et les processus de prise de décision.

BeLLA, an end-to-end architecture for autonomous driving, has been introduced, integrating unified 360° Birds Eye View representations with a large language model for enhanced question answering. This development addresses limitations in existing Vision-Language models and Multimodal Language Models by improving scene understanding and decision-making processes.

BeLLA: End-to-End Birds Eye View Large Language Assistant for Autonomous Driving

arXiv:2502.00843v2 Announce Type: replace 
Abstract: In this paper, we propose a novel approach for solving the Visual Question Answering (VQA) task in autonomous driving by integrating Vision-Language Models (VLMs) with continual learning. In autonomous driving, VQA plays a vital role in enabling the system to understand and reason about its surroundings. However, traditional models often struggle with catastrophic forgetting when sequentially exposed to new driving tasks, such as perception, prediction, and planning, each requiring different forms of knowledge. To address this challenge, we present a novel continual learning framework that combines VLMs with selective memory replay and knowledge distillation, reinforced by task-specific projection layer regularization. The knowledge distillation allows a previously trained model to act as a "teacher" to guide the model through subsequent tasks, minimizing forgetting. Meanwhile, task-specific projection layers calculate the loss based on the divergence of feature representations, ensuring continuity in learning and reducing the shift between tasks. Evaluated on the DriveLM dataset, our framework shows substantial performance improvements, with gains ranging from 20.11% to 35.16% across various metrics. These results highlight the effectiveness of combining continual learning with VLMs in enhancing the resilience and reliability of VQA systems in autonomous driving. We will release our source code.

تم اقتراح نهج جديد لمهمة الإجابة على الأسئلة المرئية (VQA) في القيادة الذاتية، من خلال دمج نماذج الرؤية-اللغة (VLM) مع تقنيات التعلم المستمر. يتناول هذا الإطار تحدي النسيان الكارثي عندما يتعرض النماذج لمهام قيادة جديدة، مما يعزز قدرتها على فهم وتفسير محيطها.

Se ha propuesto un enfoque novedoso para la respuesta a preguntas visuales (VQA) en la conducción autónoma, integrando Modelos de Visión-Lenguaje (VLM) con técnicas de aprendizaje continuo. Este marco aborda el desafío del olvido catastrófico cuando los modelos son expuestos a nuevas tareas de conducción, mejorando su capacidad para entender y razonar sobre su entorno.

Une nouvelle approche a été proposée pour la réponse à des questions visuelles (VQA) dans la conduite autonome, intégrant des modèles de vision-langage (VLM) avec des techniques d'apprentissage continu. Ce cadre aborde le défi de l'oubli catastrophique lorsque les modèles sont exposés à de nouvelles tâches de conduite, améliorant leur capacité à comprendre et à raisonner sur leur environnement.

A novel approach has been proposed for Visual Question Answering (VQA) in autonomous driving, integrating Vision-Language Models (VLMs) with continual learning techniques. This framework addresses the challenge of catastrophic forgetting when models are exposed to new driving tasks, enhancing their ability to understand and reason about their surroundings.

BeLLA: End-to-End Birds Eye View Large Language Assistant for Autonomous Driving

Was this article worth reading? Share it

LucidQuery AI

Com.locatelloapp

LangWatch