arXiv:2502.17429v3 Announce Type: replace 
Abstract: While 3D instance segmentation (3DIS) has advanced significantly, most existing methods assume that all object classes are known in advance and uniformly distributed. However, this assumption is unrealistic in dynamic, real-world environments where new classes emerge gradually and exhibit natural imbalance. Although some approaches address the emergence of new classes, they often overlook class imbalance, which leads to suboptimal performance, particularly on rare categories. To tackle this, we propose \ourmethodbf, a unified framework for \textbf{CL}ass-incremental \textbf{Imb}alance-aware \textbf{3D}IS. Building upon established exemplar replay (ER) strategies, we show that ER alone is insufficient to achieve robust performance under memory constraints. To mitigate this, we introduce a novel pseudo-label generator (PLG) that extends supervision to previously learned categories by leveraging predictions from a frozen model trained on prior tasks. Despite its promise, PLG tends to be biased towards frequent classes. Therefore, we propose a class-balanced re-weighting (CBR) scheme that estimates object frequencies from pseudo-labels and dynamically adjusts training bias, without requiring access to past data. We design and evaluate three incremental scenarios for 3DIS on the challenging ScanNet200 dataset and additionally validate our method for semantic segmentation on ScanNetV2. Our approach achieves state-of-the-art results, surpassing prior work by up to 16.76\% mAP for instance segmentation and approximately 30\% mIoU for semantic segmentation, demonstrating strong generalisation across both frequent and rare classes. Code is available at: https://github.com/vgthengane/CLIMB3D

تم اقتراح إطار عمل جديد يسمى CLIMB-3D لتحسين تقسيم الكائنات ثلاثية الأبعاد (3DIS) من خلال معالجة تحديات عدم توازن الفئات وظهور فئات جديدة من الكائنات تدريجياً في البيئات الديناميكية. تعتمد هذه الطريقة على استراتيجيات إعادة عرض النماذج الموجودة وتقدم مولدًا جديدًا للتسميات الزائفة لتحسين الأداء تحت قيود الذاكرة.

Se ha propuesto un nuevo marco llamado CLIMB-3D para mejorar la segmentación de instancias 3D (3DIS) al abordar los desafíos del desequilibrio de clases y la aparición gradual de nuevas clases de objetos en entornos dinámicos. Este enfoque se basa en estrategias de repetición de ejemplares existentes e introduce un nuevo generador de pseudoetiquetas para mejorar el rendimiento bajo restricciones de memoria.

Un nouveau cadre nommé CLIMB-3D a été proposé pour améliorer la segmentation d'instance 3D (3DIS) en abordant les défis de déséquilibre des classes et l'émergence progressive de nouvelles classes d'objets dans des environnements dynamiques. Cette approche s'appuie sur des stratégies de répétition d'exemplaires existantes et introduit un nouveau générateur de pseudo-étiquettes pour améliorer les performances sous contraintes de mémoire.

A new framework named CLIMB-3D has been proposed to enhance 3D instance segmentation (3DIS) by addressing the challenges of class imbalance and the gradual emergence of new object classes in dynamic environments. This approach builds on existing exemplar replay strategies and introduces a novel pseudo-label generator to improve performance under memory constraints.

CLIMB-3D: Continual Learning for Imbalanced 3D Instance Segmentation

arXiv:2512.07599v1 Announce Type: new 
Abstract: Online, real-time, and fine-grained 3D segmentation constitutes a fundamental capability for embodied intelligent agents to perceive and comprehend their operational environments. Recent advancements employ predefined object queries to aggregate semantic information from Vision Foundation Models (VFMs) outputs that are lifted into 3D point clouds, facilitating spatial information propagation through inter-query interactions. Nevertheless, perception is an inherently dynamic process, rendering temporal understanding a critical yet overlooked dimension within these prevailing query-based pipelines. Therefore, to further unlock the temporal environmental perception capabilities of embodied agents, our work reconceptualizes online 3D segmentation as an instance tracking problem (AutoSeg3D). Our core strategy involves utilizing object queries for temporal information propagation, where long-term instance association promotes the coherence of features and object identities, while short-term instance update enriches instant observations. Given that viewpoint variations in embodied robotics often lead to partial object visibility across frames, this mechanism aids the model in developing a holistic object understanding beyond incomplete instantaneous views. Furthermore, we introduce spatial consistency learning to mitigate the fragmentation problem inherent in VFMs, yielding more comprehensive instance information for enhancing the efficacy of both long-term and short-term temporal learning. The temporal information exchange and consistency learning facilitated by these sparse object queries not only enhance spatial comprehension but also circumvent the computational burden associated with dense temporal point cloud interactions. Our method establishes a new state-of-the-art, surpassing ESAM by 2.8 AP on ScanNet200 and delivering consistent gains on ScanNet, SceneNN, and 3RScan datasets.

أدت التطورات الأخيرة في تقسيم 3D عبر الإنترنت إلى تطوير AutoSeg3D، الذي يعيد تصور تقسيم 3D كمشكلة تتبع للكيانات. تستخدم هذه الطريقة استعلامات الكائنات لنقل المعلومات الزمنية، مما يعزز تماسك ارتباطات الكيانات على المدى الطويل في البيئات الديناميكية.

Los avances recientes en la segmentación 3D en línea han llevado al desarrollo de AutoSeg3D, que reconceptualiza la segmentación 3D como un problema de seguimiento de instancias. Este enfoque utiliza consultas de objetos para la propagación de información temporal, mejorando la coherencia de las asociaciones de instancias a largo plazo en entornos dinámicos.

Des avancées récentes dans la segmentation 3D en ligne ont conduit au développement d'AutoSeg3D, qui reconceptualise la segmentation 3D comme un problème de suivi d'instances. Cette approche utilise des requêtes d'objets pour la propagation d'informations temporelles, améliorant la cohérence des associations d'instances à long terme dans des environnements dynamiques.

Recent advancements in online 3D segmentation have led to the development of AutoSeg3D, which reconceptualizes 3D segmentation as an instance tracking problem. This approach utilizes object queries for temporal information propagation, enhancing the coherence of long-term instance associations in dynamic environments.

CLIMB-3D: Continual Learning for Imbalanced 3D Instance Segmentation

Was this article worth reading? Share it

Attentive AI

Cont3xt.dev

Dyad