arXiv:2606.13970v1 Announce Type: cross 
Abstract: Learning with missing modalities is a fundamental challenge in multimodal robot learning, as real-world robotic systems often operate in environments with incomplete sensor data. Attention-based models are appealing for processing multimodal data because they can handle multiple modalities with a single backbone network. However, most multimodal models assume that all modalities are available during both training and inference, limiting their applicability in robotic perception and decision-making. In this paper, we introduce a multimodal model designed to handle missing modalities during both training and inference. The model is formulated as a conditional variational autoencoder (CVAE) and incorporates a transformer-based architecture that leverages attention mechanisms to learn a unified, fixed-dimensional representation, even when some modalities are missing. We show that our proposed model can be trained with missing modalities while approximating a robust representation of all modalities. We evaluate our approach on five multimodal datasets across two robot learning tasks: human trajectory prediction and robot manipulation forecasting. Experimental results demonstrate that our model effectively learns from incomplete data and is superior to prior multimodal fusion approaches.

قدمت دراسة جديدة نموذجًا متعدد الوسائط يعتمد على الانتباه مصممًا للتنبؤ بشكل موثوق في السيناريوهات التي تفتقر إلى الوسائط، مما يعالج تحديًا كبيرًا في تعلم الروبوتات حيث تكون بيانات المستشعرات غير مكتملة. يستخدم هذا النموذج مشفرًا متغيرًا شرطيًا وهندسة قائمة على المحولات لإنشاء تمثيل موحد، حتى عندما تكون بعض بيانات الوسائط غائبة أثناء التدريب والاستدلال.

Un nuevo estudio ha presentado un modelo multimodal basado en atención diseñado para realizar pronósticos robustos en escenarios con modalidades faltantes, abordando un desafío significativo en el aprendizaje robótico donde los datos de sensores incompletos son comunes. Este modelo utiliza un autoencoder variacional condicional y una arquitectura de transformador para crear una representación unificada, incluso cuando algunas modalidades de datos están ausentes durante el entrenamiento y la inferencia.

Une nouvelle étude a introduit un modèle multimodal basé sur l'attention, conçu pour prévoir de manière robuste dans des scénarios avec des modalités manquantes, répondant à un défi majeur dans l'apprentissage robotique où les données de capteurs incomplètes sont courantes. Ce modèle utilise un autoencodeur variationnel conditionnel et une architecture de transformateur pour créer une représentation unifiée, même lorsque certaines modalités de données sont absentes pendant l'entraînement et l'inférence.

A new study has introduced an attention-based multimodal model designed to robustly forecast in scenarios with missing modalities, addressing a significant challenge in robotic learning where incomplete sensor data is common. This model utilizes a conditional variational autoencoder and a transformer architecture to create a unified representation, even when certain data modalities are absent during training and inference.