arXiv:2510.22370v1 Announce Type: cross 
Abstract: In this paper, we propose Bootstrapped Language-Image Pretraining-driven Fused State Representation in Proximal Policy Optimization (BLIP-FusePPO), a novel multimodal reinforcement learning (RL) framework for autonomous lane-keeping (LK), in which semantic embeddings generated by a vision-language model (VLM) are directly fused with geometric states, LiDAR observations, and Proportional-Integral-Derivative-based (PID) control feedback within the agent observation space. The proposed method lets the agent learn driving rules that are aware of their surroundings and easy to understand by combining high-level scene understanding from the VLM with low-level control and spatial signals. Our architecture brings together semantic, geometric, and control-aware representations to make policy learning more robust. A hybrid reward function that includes semantic alignment, LK accuracy, obstacle avoidance, and speed regulation helps learning to be more efficient and generalizable. Our method is different from the approaches that only use semantic models to shape rewards. Instead, it directly embeds semantic features into the state representation. This cuts down on expensive runtime inference and makes sure that semantic guidance is always available. The simulation results show that the proposed model is better at LK stability and adaptability than the best vision-based and multimodal RL baselines in a wide range of difficult driving situations. We make our code publicly available.

تم تقديم إطار جديد يسمى BLIP-FusePPO، والذي يعزز قدرات الحفاظ على المسار في المركبات المستقلة من خلال مزيج فريد من نماذج اللغة-الصورة والتعلم المعزز. يدمج هذا النهج المبتكر البيانات الدلالية والهندسية، مما قد يحسن من سلامة وكفاءة تكنولوجيا القيادة الذاتية. مع تزايد اعتماد صناعة السيارات على الأتمتة، تعتبر مثل هذه التطورات ضرورية لتطوير أنظمة موثوقة يمكنها التنقل في بيئات معقدة.

Se ha presentado un nuevo marco llamado BLIP-FusePPO, que mejora las capacidades de mantenimiento de carril en vehículos autónomos mediante una combinación única de modelos de lenguaje-visual y aprendizaje por refuerzo. Este enfoque innovador integra datos semánticos y geométricos, lo que podría mejorar la seguridad y la eficiencia de la tecnología de conducción autónoma. A medida que la industria automotriz adopta cada vez más la automatización, avances como este son cruciales para desarrollar sistemas confiables que puedan navegar en entornos complejos.

Un nouveau cadre appelé BLIP-FusePPO a été introduit, améliorant les capacités de maintien de voie dans les véhicules autonomes grâce à une combinaison unique de modèles de langage-vision et d'apprentissage par renforcement. Cette approche innovante intègre des données sémantiques et géométriques, ce qui pourrait améliorer la sécurité et l'efficacité de la technologie de conduite autonome. Alors que l'industrie automobile adopte de plus en plus l'automatisation, des avancées comme celles-ci sont cruciales pour développer des systèmes fiables capables de naviguer dans des environnements complexes.

A new framework called BLIP-FusePPO has been introduced, enhancing lane-keeping capabilities in autonomous vehicles through a unique combination of vision-language models and reinforcement learning. This innovative approach integrates semantic and geometric data, potentially improving the safety and efficiency of self-driving technology. As the automotive industry increasingly embraces automation, advancements like these are crucial for developing reliable systems that can navigate complex environments.

BLIP-FusePPO: A Vision-Language Deep Reinforcement Learning Framework for Lane Keeping in Autonomous Vehicles

arXiv:2511.17221v1 Announce Type: new 
Abstract: Learning 3D scene geometry and semantics from images is a core challenge in computer vision and a key capability for autonomous driving. Since large-scale 3D annotation is prohibitively expensive, recent work explores self-supervised learning directly from sensor data without manual labels. Existing approaches either rely on 2D rendering consistency, where 3D structure emerges only implicitly, or on discretized voxel grids from accumulated lidar point clouds, limiting spatial precision and scalability. We introduce QueryOcc, a query-based self-supervised framework that learns continuous 3D semantic occupancy directly through independent 4D spatio-temporal queries sampled across adjacent frames. The framework supports supervision from either pseudo-point clouds derived from vision foundation models or raw lidar data. To enable long-range supervision and reasoning under constant memory, we introduce a contractive scene representation that preserves near-field detail while smoothly compressing distant regions. QueryOcc surpasses previous camera-based methods by 26% in semantic RayIoU on the self-supervised Occ3D-nuScenes benchmark while running at 11.6 FPS, demonstrating that direct 4D query supervision enables strong self-supervised occupancy learning. https://research.zenseact.com/publications/queryocc/

تم تقديم QueryOcc كإطار عمل قائم على الاستقلال الذاتي للتعلم الذاتي الذي يتعلم الإشغال السيماني ثلاثي الأبعاد بشكل مستمر مباشرة من بيانات المستشعر، مما يعالج التحديات المتعلقة بالهندسة السيمانية والمشاهد ثلاثية الأبعاد في رؤية الكمبيوتر، خاصة لتطبيقات القيادة الذاتية.

Se ha presentado QueryOcc como un marco de auto-supervisión basado en consultas que aprende la ocupación semántica 3D continua directamente a partir de datos de sensores, abordando los desafíos de la geometría y la semántica de escenas 3D en la visión por computadora, especialmente para aplicaciones de conducción autónoma.

QueryOcc a été introduit comme un cadre d'auto-supervision basé sur des requêtes qui apprend l'occupation sémantique 3D continue directement à partir des données de capteurs, répondant aux défis de la géométrie et de la sémantique des scènes 3D dans la vision par ordinateur, en particulier pour les applications de conduite autonome.

QueryOcc has been introduced as a query-based self-supervised framework that learns continuous 3D semantic occupancy directly from sensor data, addressing the challenges of 3D scene geometry and semantics in computer vision, particularly for autonomous driving applications.

QueryOcc: Query-based Self-Supervision for 3D Semantic Occupancy

arXiv:2511.17455v1 Announce Type: new 
Abstract: Semantic segmentation networks trained under full supervision for one type of lidar fail to generalize to unseen lidars without intervention. To reduce the performance gap under domain shifts, a recent trend is to leverage vision foundation models (VFMs) providing robust features across domains. In this work, we conduct an exhaustive study to identify recipes for exploiting VFMs in unsupervised domain adaptation for semantic segmentation of lidar point clouds. Building upon unsupervised image-to-lidar knowledge distillation, our study reveals that: (1) the architecture of the lidar backbone is key to maximize the generalization performance on a target domain; (2) it is possible to pretrain a single backbone once and for all, and use it to address many domain shifts; (3) best results are obtained by keeping the pretrained backbone frozen and training an MLP head for semantic segmentation. The resulting pipeline achieves state-of-the-art results in four widely-recognized and challenging settings. The code will be available at: https://github.com/valeoai/muddos.

أظهرت دراسة حديثة أن شبكات تقسيم الدلالات المدربة على أنواع معينة من أجهزة الليدار تواجه صعوبة في التعميم على أنظمة الليدار الجديدة دون تدخل إضافي. تركز الدراسة على الاستفادة من نماذج الأساس البصرية (VFMs) لتعزيز التكيف غير الخاضع للإشراف لتقسيم الدلالات لنقاط الليدار، كاشفة عن رؤى معمارية رئيسية لتحسين الأداء عبر مجالات مختلفة.

Un estudio reciente ha demostrado que las redes de segmentación semántica entrenadas en tipos específicos de lidar tienen dificultades para generalizar a nuevos sistemas lidar sin intervención adicional. La investigación se centra en aprovechar los modelos de fundación visual (VFMs) para mejorar la adaptación no supervisada al dominio para la segmentación semántica de nubes de puntos lidar, revelando información arquitectónica clave para mejorar el rendimiento en diferentes dominios.

Une étude récente a montré que les réseaux de segmentation sémantique formés sur des types de lidar spécifiques ont du mal à se généraliser à de nouveaux systèmes lidar sans intervention supplémentaire. La recherche se concentre sur l'exploitation des modèles de fondation visuelle (VFMs) pour améliorer l'adaptation non supervisée au domaine pour la segmentation sémantique des nuages de points lidar, révélant des informations architecturales clés pour améliorer les performances à travers différents domaines.

A recent study has shown that semantic segmentation networks trained on specific lidar types struggle to generalize to new lidar systems without additional intervention. The research focuses on leveraging vision foundation models (VFMs) to enhance unsupervised domain adaptation for semantic segmentation of lidar point clouds, revealing key architectural insights for improving performance across different domains.

BLIP-FusePPO: A Vision-Language Deep Reinforcement Learning Framework for Lane Keeping in Autonomous Vehicles

Was this article worth reading? Share it

Republiclabs.ai

IntelliQ

The Visualizer