arXiv:2605.28490v1 Announce Type: new 
Abstract: 3D object grounding localizes referred objects in a 3D scene from natural language. Unified instance-centric 3D-LLMs aim to solve grounding together with dialog, QA, and captioning, yet many rely on a single pointer-style grounding decision that compresses a relational instruction into one selection. This is brittle for fine-grained queries where multiple same-class candidates must be ruled out by context objects and spatial relations. We propose Structured Spatial Reasoning 3D-LLM (SSR3D-LLM), a structured grounding interface for unified 3D-LLMs. Given fixed Mask3D object proposals, the LLM writes a sequence of latent spatial reasoning steps and memory tokens from the query, and a geometry-aware scorer reads these latent steps in order to refine candidate rankings step by step with step-length masking. The latent steps are learned from standard benchmark target supervision with auxiliary referential-cue supervision during training, while inference uses only the input query and Mask3D proposals. Across ReferIt3D, ScanRefer, and Multi3DRef, SSR3D-LLM achieves the strongest results among unified 3D-LLM baselines, with substantial gains over the single-pointer QPG baseline on fine-grained grounding and consistent improvements over prior unified 3D-LLMs, while preserving the default language-task route.

تم تقديم SSR3D-LLM، أو التفكير المكاني المنظم عبر خطوات كامنة، كنهج جديد لتحسين تحديد الكائنات ثلاثية الأبعاد من خلال واجهة تحديد منظمة تكتب خطوات التفكير المكاني الكامنة، مما يحسن تحديد الكائنات المشار إليها في مشاهد ثلاثية الأبعاد من استفسارات اللغة الطبيعية.

Se ha introducido el SSR3D-LLM, o Razonamiento Espacial Estructurado a través de Pasos Latentes, como un enfoque novedoso para mejorar la localización de objetos en 3D mediante una interfaz de localización estructurada que escribe pasos de razonamiento espacial latentes, mejorando así la localización de objetos referidos en escenas 3D a partir de consultas en lenguaje natural.

Le SSR3D-LLM, ou Raisonnement Spatial Structuré via des Étapes Latentes, a été introduit comme une nouvelle approche pour améliorer le repérage d'objets en 3D en utilisant une interface de repérage structurée qui écrit des étapes de raisonnement spatial latentes, améliorant ainsi la localisation des objets référencés dans des scènes 3D à partir de requêtes en langage naturel.

The Structured Spatial Reasoning 3D-LLM (SSR3D-LLM) has been introduced as a novel approach to enhance 3D object grounding by utilizing a structured grounding interface that writes latent spatial reasoning steps, improving the localization of referred objects in 3D scenes from natural language queries.

SSR3D-LLM: Structured Spatial Reasoning via Latent Steps for Fine-Grained Grounding in Unified 3D-LLMs

arXiv:2606.10594v1 Announce Type: new 
Abstract: 3D vision-language segmentation aims to segment target objects in 3D scenarios according to the linguistic instructions and visual observations. Prior art heavily relies on the coarse superpoint representation to reduce the computation complexity, which suffers from poor segmentation quality and messy object boundaries. In this paper, we propose the SEGment-And-select (SEGA3D) paradigm for 3D visionlanguage segmentation that directly operates on the fine-grained visual information and is free from the superpoint dependency. Specifically, we first leverage a mask candidate generator to provide fine-grained categorical mask candidates, substantially improving the quality of candidate masks over the superpoint counterparts. Then, a Large Language Model (LLM) is utilized to generate the semantic and spatial information based on the linguistic description and visual features. The LLM output and visual features are fed to the Semantic-Spatial Selector (SSS) to produce the top-ranking mask candidates. Eventually, the Loopback Verification Module (LVM) is designed to yield the segmentation mask from the selected candidate masks. Our SEGA3D attains competitive performance on ScanRefer, ScanNet and Matterport3D benchmarks. Notably, our SEGA3D surpasses the top-performing counterpart by 8.3 mIoU and 5.3 mIoU on ScanNet and Matterport3D, respectively. Codes will be available upon publication.

يمثل تقديم نموذج SEGment-And-select (SEGA3D) مؤخرًا تقدمًا كبيرًا في مجال تقسيم الرؤية-اللغة في السيناريوهات ثلاثية الأبعاد، مما يسمح بتحسين تقسيم الأجسام في السيناريوهات ثلاثية الأبعاد وفقًا للتعليمات اللغوية والملاحظات البصرية. تلغي هذه الطريقة الاعتماد على تمثيلات النقاط الخشنة، مما يعزز جودة التقسيم من خلال المعلومات البصرية الدقيقة.

La reciente introducción del paradigma SEGment-And-select (SEGA3D) marca un avance significativo en la segmentación de visión-lenguaje en 3D, permitiendo una mejor segmentación de objetos en escenarios 3D según instrucciones lingüísticas y observaciones visuales. Este enfoque elimina la dependencia de representaciones superpoint gruesas, mejorando la calidad de la segmentación a través de información visual detallada.

L'introduction récente du paradigme SEGment-And-select (SEGA3D) représente une avancée significative dans la segmentation vision-langage en 3D, permettant une meilleure segmentation des objets dans des scénarios 3D en fonction des instructions linguistiques et des observations visuelles. Cette approche élimine la dépendance aux représentations superpoint grossières, améliorant ainsi la qualité de la segmentation grâce à des informations visuelles fines.

The recent introduction of the SEGment-And-select (SEGA3D) paradigm marks a significant advancement in 3D vision-language segmentation, allowing for improved object segmentation in 3D scenarios based on linguistic instructions and visual observations. This approach eliminates reliance on coarse superpoint representations, enhancing segmentation quality through fine-grained visual information.

SSR3D-LLM: Structured Spatial Reasoning via Latent Steps for Fine-Grained Grounding in Unified 3D-LLMs

Was this article worth reading? Share it

Ready to build your own newsroom?