arXiv:2506.22242v2 Announce Type: replace 
Abstract: Leveraging diverse robotic data for pretraining remains a critical challenge. Existing methods typically model the dataset's action distribution using simple observations as inputs. However, these inputs are often incomplete, resulting in a dispersed conditional action distribution-an issue we refer to as coordinate system chaos and state chaos. This inconsistency significantly hampers pretraining efficiency. To address this, we propose 4D-VLA, a novel approach that effectively integrates 4D information into the input to mitigate these sources of chaos. Our model introduces depth and temporal information into visual features with sequential RGB-D inputs, aligning the coordinate systems of the robot and the scene. This alignment endows the model with strong spatiotemporal reasoning capabilities while minimizing training overhead. Additionally, we introduce memory bank sampling, a frame sampling strategy designed to extract informative frames from historical images, further improving effectiveness and efficiency. Experimental results demonstrate that our pretraining method and architectural components substantially enhance model performance. In both simulated and real-world experiments, our model achieves a significant increase in success rate over OpenVLA. To further assess spatial perception and generalization to novel views, we introduce MV-Bench, a multi-view simulation benchmark. Our model consistently outperforms existing methods, demonstrating stronger spatial understanding and adaptability.

يتناول المقال المعنون '4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration' التحديات المتعلقة بالاستفادة من بيانات الروبوتات المتنوعة للتدريب المسبق. تعتمد الطرق الحالية غالبًا على ملاحظات غير مكتملة، مما يؤدي إلى توزيع غير منظم للإجراءات. النموذج المقترح، 4D-VLA، يدمج معلومات العمق والزمن في الميزات البصرية باستخدام مدخلات RGB-D متسلسلة، مما ينسق نظام إحداثيات الروبوت مع نظام إحداثيات المشهد. هذه الطريقة تعزز قدرات التفكير الزمني المكاني مع تقليل عبء التدريب.

El artículo titulado '4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration' aborda los desafíos de aprovechar datos robóticos diversos para el preentrenamiento. Los métodos actuales suelen basarse en observaciones incompletas, lo que conduce a una distribución de acciones caótica. El modelo propuesto, 4D-VLA, integra información de profundidad y temporal en características visuales utilizando entradas RGB-D secuenciales, alineando el sistema de coordenadas del robot con el de la escena. Este enfoque mejora las capacidades de razonamiento espaciotemporal mientr…

L'article intitulé '4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration' traite des défis liés à l'utilisation de données robotiques diverses pour le préentraînement. Les méthodes actuelles reposent souvent sur des observations incomplètes, entraînant une distribution d'actions chaotique. Le modèle proposé, 4D-VLA, intègre des informations de profondeur et temporelles dans les caractéristiques visuelles à l'aide d'entrées RGB-D séquentielles, alignant le système de coordonnées du robot avec celui de la scène. Cette approche améliore les capacités de raisonnem…

The paper titled '4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration' addresses the challenges of leveraging diverse robotic data for pretraining. Current methods often rely on incomplete observations, leading to a chaotic action distribution. The proposed 4D-VLA model integrates depth and temporal information into visual features using sequential RGB-D inputs, aligning the robot's coordinate system with that of the scene. This approach enhances spatiotemporal reasoning capabilities while reducing training overhead.

4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration

One More Thing in AI – Your Shortcut to AI Mastery

4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

The Visualizer

Dyad

Attentive AI

Videolulu

Ready to build your own newsroom?