arXiv:2605.26831v1 Announce Type: new 
Abstract: Semantic mapping methods are increasingly used as intermediate scene representations for downstream robotic reasoning and manipulation, yet their evaluation is still largely tied to fixed benchmark datasets with limited coverage of manipulation-relevant corner cases. In this work, we extend OSMa-Bench toward controllable benchmarking with prompt-generated synthetic indoor scenes. Our pipeline automatically generates scene descriptions, synthesizes corresponding environments with SceneSmith, and adapts the resulting assets into an OSMa-Bench-compatible simulation format. This adaptation requires a nontrivial intermediate layer, including semantic normalization, material and texture repair, shader fallback policies, floor handling, navigation setup, and controlled lighting configuration. A key advantage of the proposed setup is that the original scene-generation prompt is known in advance and can therefore serve as an auxiliary semantic specification of the intended scene. We use this property to extend the VQA component of OSMa-Bench with a prompt-grounded question category. The resulting framework supports targeted stress-testing of semantic scene representations under conditions such as clutter, small objects, partial occlusions, and lighting variation, and makes benchmarking more extensible and better aligned with downstream manipulation requirements. Our code is available at https://github.com/be2rlab/OSMa-Bench-v2.

التطور الأخير لـ OSMa-Bench++ يهدف إلى تحسين تقييم طرق الخرائط الدلالية المستخدمة في المناورة الروبوتية من خلال تقديم تقييم قابل للتحكم مع مشاهد داخلية اصطناعية تم إنشاؤها بواسطة مطالبات. تتيح هذه الطريقة إنشاء أوصاف مشهد تلقائيًا وتوليد بيئات متوافقة، والتي يتم تكييفها بعد ذلك في تنسيق محاكاة متوافق مع OSMa-Bench.

El reciente desarrollo de OSMa-Bench++ tiene como objetivo mejorar la evaluación de los métodos de mapeo semántico utilizados en la manipulación robótica al introducir un benchmarking controlable con escenas interiores sintéticas generadas por indicaciones. Este enfoque permite la generación automática de descripciones de escenas y la síntesis de entornos correspondientes, que luego se adaptan a un formato de simulación compatible con OSMa-Bench.

Le développement récent d'OSMa-Bench++ vise à améliorer l'évaluation des méthodes de cartographie sémantique utilisées dans la manipulation robotique en introduisant un benchmarking contrôlable avec des scènes intérieures synthétiques générées par des invites. Cette approche permet la génération automatique de descriptions de scènes et la synthèse d'environnements correspondants, qui sont ensuite adaptés dans un format de simulation compatible avec OSMa-Bench.

The recent development of OSMa-Bench++ aims to enhance the evaluation of semantic mapping methods used in robotic manipulation by introducing controllable benchmarking with prompt-generated synthetic indoor scenes. This approach allows for the automatic generation of scene descriptions and the synthesis of corresponding environments, which are then adapted into a simulation format compatible with OSMa-Bench.

OSMa-Bench++: Toward Open-Ended Benchmarking of Semantic Mapping for Manipulation with Prompt-Generated Synthetic Scenes

arXiv:2606.08063v1 Announce Type: new 
Abstract: Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in visual understanding, yet their performance degrades significantly under real-world visual corruptions. While existing robustness enhancement approaches exist, they are limited: black-box feature alignment lacks interpretability, and white-box text-based reasoning cannot restore lost pixel-level details. This work investigates a fundamental research question: Can MLLMs recover corrupted visual content by themselves? To address this, we propose Robust-U1, a novel framework that equips MLLMs with explicit visual self-recovery capability for robust understanding. The approach comprises three core stages: supervised fine-tuning for initial reconstruction, reinforcement learning with dual rewards (pixel-level SSIM and semantic-level CLIP similarity) for aligning high visual quality, and multimodal reasoning that jointly considers both the corrupted input and the recovered image. Extensive experiments demonstrate that Robust-U1 achieves state-of-the-art robustness on the real-world corruption benchmark and maintains superior performance under adversarial corruptions on general VQA benchmarks. Analysis confirms that high-quality visual recovery directly enhances reasoning performance, establishing self-recovery as a critical mechanism for robust visual understanding. The source code is available at https://github.com/jqtangust/Robust-U1.

تمثل مقدمة Robust-U1 تقدمًا كبيرًا في قدرات نماذج اللغة متعددة الوسائط الكبيرة (MLLMs)، مع التركيز على قدرتها على استعادة المحتوى المرئي التالف لفهم أفضل. يستخدم هذا الإطار التعديل الخاضع للإشراف، والتعلم المعزز، والتفكير متعدد الوسائط لتحسين الجودة المرئية والمحاذاة الدلالية.

La introducción de Robust-U1 marca un avance significativo en las capacidades de los Modelos de Lenguaje Multimodal de Gran Escala (MLLMs), centrándose en su capacidad para auto-recuperar contenido visual corrupto para una mejor comprensión. Este marco emplea ajuste supervisado, aprendizaje por refuerzo y razonamiento multimodal para mejorar la calidad visual y la alineación semántica.

L'introduction de Robust-U1 représente une avancée significative dans les capacités des Modèles de Langage Multimodal de Grande Taille (MLLMs), en mettant l'accent sur leur capacité à se rétablir eux-mêmes d'un contenu visuel corrompu pour une compréhension améliorée. Ce cadre utilise un ajustement supervisé, un apprentissage par renforcement et un raisonnement multimodal pour améliorer la qualité visuelle et l'alignement sémantique.

The introduction of Robust-U1 marks a significant advancement in the capabilities of Multimodal Large Language Models (MLLMs), focusing on their ability to self-recover corrupted visual content for enhanced understanding. This framework employs supervised fine-tuning, reinforcement learning, and multimodal reasoning to improve visual quality and semantic alignment.

OSMa-Bench++: Toward Open-Ended Benchmarking of Semantic Mapping for Manipulation with Prompt-Generated Synthetic Scenes

Was this article worth reading? Share it

Ready to build your own newsroom?