arXiv:2511.11266v2 Announce Type: replace 
Abstract: Vision-language models have recently emerged as promising planners for autonomous driving, where success hinges on topology-aware reasoning over spatial structure and dynamic interactions from multimodal input. However, existing models are typically trained without supervision that explicitly encodes these relational dependencies, limiting their ability to infer how agents and other traffic entities influence one another from raw sensor data. In this work, we bridge this gap with a novel model-agnostic method that conditions language-based driving models on structured relational context in the form of traffic scene graphs. We serialize scene graphs at various abstraction levels and formats, and incorporate them into the models via structured prompt templates, enabling a systematic analysis of when and how relational supervision is most beneficial. Extensive evaluations on the public LangAuto benchmark show that scene graph conditioning of state-of-the-art approaches yields large and persistent improvement in driving performance. Notably, we observe up to a 15.6\% increase in driving score for LMDrive and 17.5\% for BEVDriver, indicating that models can better internalize and ground relational priors through scene graph-conditioned training, even without requiring scene graph input at test-time. Code, fine-tuned models, and our scene graph dataset are publicly available at https://github.com/iis-esslingen/GraphPilot.

يقدم GraphPilot طريقة جديدة غير مرتبطة بالنموذج تعزز القيادة الذاتية المعتمدة على اللغة من خلال تكييف النماذج على سياق علاقاتي منظم عبر رسوم بيانية لمشاهد المرور. تتناول هذه الطريقة قيود النماذج الحالية التي تفتقر إلى إشراف علاقاتي صريح، مما يحسن قدرتها على تفسير التفاعلات الديناميكية من المدخلات متعددة الوسائط.

GraphPilot presenta un nuevo método agnóstico al modelo que mejora la conducción autónoma basada en lenguaje al condicionar los modelos en un contexto relacional estructurado a través de gráficos de escenas de tráfico. Este enfoque aborda las limitaciones de los modelos existentes que carecen de supervisión relacional explícita, mejorando así su capacidad para interpretar interacciones dinámicas a partir de entradas multimodales.

GraphPilot introduit une nouvelle méthode agnostique au modèle qui améliore la conduite autonome basée sur le langage en conditionnant les modèles sur un contexte relationnel structuré à travers des graphes de scènes de trafic. Cette approche répond aux limites des modèles existants qui manquent de supervision relationnelle explicite, améliorant ainsi leur capacité à interpréter les interactions dynamiques à partir d'entrées multimodales.

GraphPilot introduces a novel model-agnostic method that enhances language-based autonomous driving by conditioning models on structured relational context through traffic scene graphs. This approach addresses the limitations of existing models that lack explicit relational supervision, thereby improving their ability to interpret dynamic interactions from multimodal inputs.

GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving

One More Thing in AI – Your Shortcut to AI Mastery

GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

Guidejar-4eb95b

Attentive AI

LangWatch

Graphite Note

Ready to build your own newsroom?