arXiv:2506.22242v2 Announce Type: replace 
Abstract: Leveraging diverse robotic data for pretraining remains a critical challenge. Existing methods typically model the dataset's action distribution using simple observations as inputs. However, these inputs are often incomplete, resulting in a dispersed conditional action distribution-an issue we refer to as coordinate system chaos and state chaos. This inconsistency significantly hampers pretraining efficiency. To address this, we propose 4D-VLA, a novel approach that effectively integrates 4D information into the input to mitigate these sources of chaos. Our model introduces depth and temporal information into visual features with sequential RGB-D inputs, aligning the coordinate systems of the robot and the scene. This alignment endows the model with strong spatiotemporal reasoning capabilities while minimizing training overhead. Additionally, we introduce memory bank sampling, a frame sampling strategy designed to extract informative frames from historical images, further improving effectiveness and efficiency. Experimental results demonstrate that our pretraining method and architectural components substantially enhance model performance. In both simulated and real-world experiments, our model achieves a significant increase in success rate over OpenVLA. To further assess spatial perception and generalization to novel views, we introduce MV-Bench, a multi-view simulation benchmark. Our model consistently outperforms existing methods, demonstrating stronger spatial understanding and adaptability.

يتناول المقال المعنون '4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration' التحديات المتعلقة بالاستفادة من بيانات الروبوتات المتنوعة للتدريب المسبق. تعتمد الطرق الحالية غالبًا على ملاحظات غير مكتملة، مما يؤدي إلى توزيع غير منظم للإجراءات. النموذج المقترح، 4D-VLA، يدمج معلومات العمق والزمن في الميزات البصرية باستخدام مدخلات RGB-D متسلسلة، مما ينسق نظام إحداثيات الروبوت مع نظام إحداثيات المشهد. هذه الطريقة تعزز قدرات التفكير الزمني المكاني مع تقليل عبء التدريب.

El artículo titulado '4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration' aborda los desafíos de aprovechar datos robóticos diversos para el preentrenamiento. Los métodos actuales suelen basarse en observaciones incompletas, lo que conduce a una distribución de acciones caótica. El modelo propuesto, 4D-VLA, integra información de profundidad y temporal en características visuales utilizando entradas RGB-D secuenciales, alineando el sistema de coordenadas del robot con el de la escena. Este enfoque mejora las capacidades de razonamiento espaciotemporal mientr…

L'article intitulé '4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration' traite des défis liés à l'utilisation de données robotiques diverses pour le préentraînement. Les méthodes actuelles reposent souvent sur des observations incomplètes, entraînant une distribution d'actions chaotique. Le modèle proposé, 4D-VLA, intègre des informations de profondeur et temporelles dans les caractéristiques visuelles à l'aide d'entrées RGB-D séquentielles, alignant le système de coordonnées du robot avec celui de la scène. Cette approche améliore les capacités de raisonnem…

The paper titled '4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration' addresses the challenges of leveraging diverse robotic data for pretraining. Current methods often rely on incomplete observations, leading to a chaotic action distribution. The proposed 4D-VLA model integrates depth and temporal information into visual features using sequential RGB-D inputs, aligning the robot's coordinate system with that of the scene. This approach enhances spatiotemporal reasoning capabilities while reducing training overhead.

4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration

arXiv:2511.16203v1 Announce Type: new 
Abstract: Vision-Language-Action models (VLAs) have recently demonstrated remarkable progress in embodied environments, enabling robots to perceive, reason, and act through unified multimodal understanding. Despite their impressive capabilities, the adversarial robustness of these systems remains largely unexplored, especially under realistic multimodal and black-box conditions. Existing studies mainly focus on single-modality perturbations and overlook the cross-modal misalignment that fundamentally affects embodied reasoning and decision-making. In this paper, we introduce VLA-Fool, a comprehensive study of multimodal adversarial robustness in embodied VLA models under both white-box and black-box settings. VLA-Fool unifies three levels of multimodal adversarial attacks: (1) textual perturbations through gradient-based and prompt-based manipulations, (2) visual perturbations via patch and noise distortions, and (3) cross-modal misalignment attacks that intentionally disrupt the semantic correspondence between perception and instruction. We further incorporate a VLA-aware semantic space into linguistic prompts, developing the first automatically crafted and semantically guided prompting framework. Experiments on the LIBERO benchmark using a fine-tuned OpenVLA model reveal that even minor multimodal perturbations can cause significant behavioral deviations, demonstrating the fragility of embodied multimodal alignment.

أظهرت نماذج اللغة-الرؤية-العمل (VLA) تقدمًا ملحوظًا في البيئات المجسدة، مما يمكّن الروبوتات من الإدراك والتفكير والعمل من خلال فهم متعدد الوسائط موحد. ومع ذلك، لا تزال قوة هذه الأنظمة ضد الهجمات العدائية غير مستكشفة إلى حد كبير، خاصة في ظل ظروف متعددة الوسائط وبيئات مغلقة. يقدم هذا المقال VLA-Fool، وهو دراسة تركز على قوة الهجمات العدائية متعددة الوسائط في نماذج VLA، مع معالجة مشكلات مثل الاضطرابات النصية والمرئية وعدم التوافق بين الوسائط.

Los modelos de Acción-Lenguaje-Visión (VLA) han mostrado avances significativos en entornos incorporados, permitiendo a los robots percibir, razonar y actuar a través de una comprensión multimodal unificada. Sin embargo, la robustez adversarial de estos sistemas sigue siendo poco explorada, especialmente en condiciones multimodales realistas y de caja negra. Este artículo presenta VLA-Fool, un estudio que se centra en la robustez adversarial multimodal en modelos VLA, abordando problemas como perturbaciones textuales y visuales y desalineación intermodal.

Les modèles Vision-Language-Action (VLA) ont montré des avancées significatives dans des environnements incarnés, permettant aux robots de percevoir, raisonner et agir grâce à une compréhension multimodale unifiée. Cependant, leur robustesse face aux attaques adversariales reste peu explorée, en particulier dans des scénarios multimodaux réalistes et en boîte noire. Cet article présente VLA-Fool, une étude axée sur la robustesse adversariale multimodale des VLA, abordant des problèmes tels que les perturbations textuelles et visuelles et le désalignement intermodal.

Vision-Language-Action models (VLAs) have shown significant advancements in embodied environments, allowing robots to perceive, reason, and act through a unified multimodal understanding. However, their adversarial robustness remains under-researched, particularly in realistic multimodal and black-box scenarios. This paper introduces VLA-Fool, a study focusing on multimodal adversarial robustness in VLAs, addressing issues like textual and visual perturbations and cross-modal misalignment.

4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration

Was this article worth reading? Share it