arXiv:2511.16264v1 Announce Type: new 
Abstract: Realistic and smooth full-body tracking is crucial for immersive AR/VR applications. Existing systems primarily track head and hands via Head Mounted Devices (HMDs) and controllers, making the 3D full-body reconstruction in-complete. One potential approach is to generate the full-body motions from sparse inputs collected from limited sensors using a Neural Network (NN) model. In this paper, we propose a novel method based on a multi-layer perceptron (MLP) backbone that is enhanced with residual connections and a novel NN-component called Memory-Block. In particular, Memory-Block represents missing sensor data with trainable code-vectors, which are combined with the sparse signals from previous time instances to improve the temporal consistency. Furthermore, we formulate our solution as a multi-task learning problem, allowing our MLP-backbone to learn robust representations that boost accuracy. Our experiments show that our method outperforms state-of-the-art baselines by substantially reducing prediction errors. Moreover, it achieves 72 FPS on mobile HMDs that ultimately improves the accuracy-running time tradeoff.

تقدم الورقة Mem-MLP، وهي طريقة جديدة لتوليد حركة الإنسان ثلاثية الأبعاد في الوقت الحقيقي من مدخلات نادرة باستخدام نموذج متعدد الطبقات من البيرسيترون (MLP). تتناول هذه الطريقة قيود أنظمة الواقع المعزز/الواقع الافتراضي التي تتعقب بشكل أساسي الرأس واليدين، مما يؤدي إلى إعادة بناء جسم غير مكتملة. يتضمن Mem-MLP كتلة ذاكرة لتمثيل بيانات المستشعرات المفقودة ويحسن الاتساق الزمني من خلال التعلم متعدد المهام، مما يزيد من دقة تتبع الحركة.

El artículo presenta Mem-MLP, un método novedoso para la generación de movimiento humano 3D en tiempo real a partir de entradas escasas, utilizando un modelo de perceptrón multicapa (MLP). Este enfoque aborda las limitaciones de los sistemas AR/VR que rastrean principalmente la cabeza y las manos, lo que resulta en reconstrucciones corporales incompletas. Mem-MLP incorpora un Memory-Block para representar datos de sensores faltantes y mejora la consistencia temporal a través del aprendizaje multitarea, aumentando la precisión en el seguimiento del movimiento.

Cet article présente Mem-MLP, une méthode novatrice pour la génération de mouvements humains 3D en temps réel à partir d'entrées rares, utilisant un modèle de perceptron multicouche (MLP). Cette approche répond aux limites des systèmes AR/VR qui suivent principalement la tête et les mains, entraînant des reconstructions corporelles incomplètes. Mem-MLP intègre un Memory-Block pour représenter les données de capteurs manquantes et améliore la cohérence temporelle grâce à l'apprentissage multitâche, augmentant ainsi la précision du suivi des mouvements.

The paper presents Mem-MLP, a novel method for real-time 3D human motion generation from sparse inputs using a multi-layer perceptron (MLP) model. This approach addresses the limitations of existing AR/VR systems that primarily track head and hands, resulting in incomplete full-body reconstructions. Mem-MLP incorporates a Memory-Block to represent missing sensor data and improves temporal consistency through multi-task learning, enhancing accuracy in motion tracking.

Mem-MLP: Real-Time 3D Human Motion Generation from Sparse Inputs

arXiv:2503.09878v3 Announce Type: replace 
Abstract: Vision foundation models (VFMs) such as DINO have led to a paradigm shift in 2D camera-based perception towards extracting generalized features to support many downstream tasks. Recent works introduce self-supervised cross-modal knowledge distillation (KD) as a way to transfer these powerful generalization capabilities into 3D LiDAR-based models. However, they either rely on highly complex distillation losses, pseudo-semantic maps, or limit KD to features useful for semantic segmentation only. In this work, we propose CleverDistiller, a self-supervised, cross-modal 2D-to-3D KD framework introducing a set of simple yet effective design choices: Unlike contrastive approaches relying on complex loss design choices, our method employs a direct feature similarity loss in combination with a multi layer perceptron (MLP) projection head to allow the 3D network to learn complex semantic dependencies throughout the projection. Crucially, our approach does not depend on pseudo-semantic maps, allowing for direct knowledge transfer from a VFM without explicit semantic supervision. Additionally, we introduce the auxiliary self-supervised spatial task of occupancy prediction to enhance the semantic knowledge, obtained from a VFM through KD, with 3D spatial reasoning capabilities. Experiments on standard autonomous driving benchmarks for 2D-to-3D KD demonstrate that CleverDistiller achieves state-of-the-art performance in both semantic segmentation and 3D object detection (3DOD) by up to 10% mIoU, especially when fine tuning on really low data amounts, showing the effectiveness of our simple yet powerful KD strategy

CleverDistiller هو إطار لتقطير المعرفة عبر الأنماط ذاتية الإشراف، يعزز نقل الميزات من نماذج الرؤية ثنائية الأبعاد إلى نماذج LiDAR ثلاثية الأبعاد. يبسط عملية التقطير باستخدام خسارة مباشرة لتشابه الميزات ورأس إسقاط متعدد الطبقات، مما يسمح بتعلم أفضل للاعتماديات الدلالية المعقدة. تهدف هذه الطريقة إلى تحسين أداء النماذج ثلاثية الأبعاد في تطبيقات متنوعة، بما في ذلك القيادة الذاتية.

CleverDistiller es un marco de destilación de conocimientos cross-modal auto-supervisado que mejora la transferencia de características de modelos de visión 2D a modelos LiDAR 3D. Simplifica el proceso de destilación utilizando una pérdida de similitud de características directa y una cabeza de proyección de perceptrón multicapa, permitiendo un mejor aprendizaje de dependencias semánticas complejas. Este enfoque busca mejorar el rendimiento de los modelos 3D en diversas aplicaciones, incluida la conducción autónoma.

CleverDistiller est un cadre d'évaporation de connaissances cross-modal auto-supervisé qui améliore le transfert de caractéristiques des modèles de vision 2D vers les modèles LiDAR 3D. Il simplifie le processus d'évaporation en utilisant une perte de similarité de caractéristiques directe et une tête de projection à perceptron multicouche, permettant un meilleur apprentissage des dépendances sémantiques complexes. Cette approche vise à améliorer les performances des modèles 3D dans diverses applications, y compris la conduite autonome.

CleverDistiller is a self-supervised, cross-modal knowledge distillation framework that enhances the transfer of features from 2D vision models to 3D LiDAR models. It simplifies the distillation process by using a direct feature similarity loss and a multi-layer perceptron projection head, allowing for better learning of complex semantic dependencies. This approach aims to improve the performance of 3D models in various applications, including autonomous driving.

CleverDistiller: Simple and Spatially Consistent Cross-modal Distillation

arXiv:2511.15343v1 Announce Type: cross 
Abstract: Developing reliable UAV navigation systems requires robust air-to-air object detectors capable of distinguishing between objects seen during training and previously unseen objects. While many methods address closed-set detection and achieve high-confidence recognition of in-domain (ID) targets, they generally do not tackle open-set detection, which requires simultaneous handling of both ID and out-of-distribution (OOD) objects. Existing open-set approaches typically rely on a single uncertainty score with thresholding, limiting flexibility and often conflating OOD objects with background clutter. In contrast, we propose a lightweight, model-agnostic post-processing framework that explicitly separates background from unknown objects while preserving the base detector's performance. Our approach extends open-set detection beyond binary ID/OOD classification to real-time three-way classification among ID targets, OOD objects, and background. To this end, we employ a fusion scheme that aggregates multiple confidence estimates and per-detection features using a compact multilayer perceptron (MLP). Incorporating different logit variants into the MLP consistently enhances performance across both binary and three-class classification without compromising throughput. Extensive ablation and comparative experiments confirm that our method surpasses threshold-based baselines in two-class classification by an average of 2.7% AUROC, while retaining or improving open-set mAP. Furthermore, our study uniquely enables robust three-class classification, a critical capability for safe UAV navigation, where OOD objects must be actively avoided and background regions safely ignored. Comparative analysis highlights that our method surpasses competitive techniques in AUROC across datasets, while improving closed-set mAP by up to 9 points, an 18% relative gain.

يتناول المقال إطارًا جديدًا للكشف عن الأجسام الجوية في بيئة مفتوحة باستخدام الطائرات بدون طيار (UAV). يتناول القيود المفروضة على الأساليب الحالية التي تركز بشكل أساسي على الكشف في بيئة مغلقة، والتي لا تميز بشكل فعال بين الأجسام المعروفة وغير المعروفة. يعزز النهج المقترح، المستقل عن النموذج، قدرات الكشف من خلال تمكين التصنيف في الوقت الحقيقي للأهداف المعروفة، والأجسام خارج التوزيع، والخلفية، مما يحسن موثوقية أنظمة الملاحة للطائرات بدون طيار.

El artículo presenta un nuevo marco para la detección de objetos aéreos en un entorno abierto utilizando UAVs. Aborda las limitaciones de los métodos existentes que se centran principalmente en la detección en un entorno cerrado, que no distinguen eficazmente entre objetos conocidos y desconocidos. El enfoque propuesto, independiente del modelo, mejora las capacidades de detección al permitir la clasificación en tiempo real de objetivos en dominio, objetos fuera de distribución y fondo, mejorando así la fiabilidad de los sistemas de navegación de UAV.

L'article présente un nouveau cadre pour la détection d'objets aériens en open-set utilisant des UAV. Il aborde les limites des méthodes existantes qui se concentrent principalement sur la détection en closed-set, ne distinguant pas efficacement les objets connus des objets inconnus. L'approche proposée, agnostique au modèle, améliore les capacités de détection en permettant une classification en temps réel des cibles in-domain, des objets hors distribution et de l'arrière-plan, améliorant ainsi la fiabilité des systèmes de navigation UAV.

The article discusses a new framework for open-set aerial object detection using UAVs. It addresses the limitations of existing methods that primarily focus on closed-set detection, which do not effectively distinguish between known and unknown objects. The proposed model-agnostic approach enhances detection capabilities by enabling real-time classification of in-domain targets, out-of-distribution objects, and background, thereby improving the reliability of UAV navigation systems.

Mem-MLP: Real-Time 3D Human Motion Generation from Sparse Inputs

Was this article worth reading? Share it