arXiv:2602.03890v3 Announce Type: replace 
Abstract: Point clouds provide a compact and expressive representation of 3D objects, and have recently been integrated into multimodal large language models (MLLMs). However, existing methods primarily focus on static objects, while understanding dynamic point cloud sequences remains largely unexplored. This limitation is mainly caused by the lack of large-scale cross-modal datasets and the difficulty of modeling motions in spatio-temporal contexts. To bridge this gap, we present 4DPC$^2$hat, the first MLLM tailored for dynamic point cloud understanding. To this end, we construct a large-scale cross-modal dataset 4DPC$^2$hat-200K via a meticulous two-stage pipeline consisting of topology-consistent 4D point construction and two-level captioning. The dataset contains over 44K dynamic object sequences, 700K point cloud frames, and 200K curated question-answer (QA) pairs, supporting inquiries about counting, temporal relationship, action, spatial relationship, and appearance. At the core of the framework, we introduce a Mamba-enhanced temporal reasoning MLLM to capture long-range dependencies and dynamic patterns among a point cloud sequence. Furthermore, we propose a failure-aware bootstrapping learning strategy that iteratively identifies model deficiencies and generates targeted QA supervision to continuously strengthen corresponding reasoning capabilities. Extensive experiments demonstrate that our 4DPC$^2$hat significantly improves action understanding and temporal reasoning compared with existing models, establishing a strong foundation for 4D dynamic point cloud understanding.

يمثل تقديم 4DPC$^2$hat تقدمًا كبيرًا في فهم سحب النقاط الديناميكية، حيث يعالج فجوة في نماذج اللغة متعددة الوسائط (MLLMs) الموجودة التي تركز بشكل أساسي على الأجسام الثابتة. يعتمد هذا النموذج الجديد على إنشاء مجموعة بيانات 4DPC$^2$hat-200K، التي تتضمن أكثر من 44,000 تسلسل كائن ديناميكي و200,000 زوج من الأسئلة والأجوبة المنسقة بعناية.

La introducción de 4DPC$^2$hat marca un avance significativo en la comprensión de nubes de puntos dinámicas, abordando una brecha en los modelos de lenguaje multimodal (MLLMs) existentes que se centran principalmente en objetos estáticos. Este nuevo modelo se apoya en la creación del conjunto de datos 4DPC$^2$hat-200K, que incluye más de 44,000 secuencias de objetos dinámicos y 200,000 pares de preguntas y respuestas cuidadosamente seleccionadas.

L'introduction de 4DPC$^2$hat représente une avancée significative dans la compréhension des nuages de points dynamiques, comblant une lacune dans les modèles de langage multimodaux (MLLMs) existants qui se concentrent principalement sur des objets statiques. Ce nouveau modèle est soutenu par la création du jeu de données 4DPC$^2$hat-200K, qui comprend plus de 44 000 séquences d'objets dynamiques et 200 000 paires de questions-réponses soigneusement élaborées.

The introduction of 4DPC$^2$hat marks a significant advancement in the understanding of dynamic point clouds, addressing a gap in existing multimodal large language models (MLLMs) that primarily focus on static objects. This new model is supported by the creation of the 4DPC$^2$hat-200K dataset, which includes over 44,000 dynamic object sequences and 200,000 curated question-answer pairs.