arXiv:2511.12878v3 Announce Type: replace 
Abstract: Forecasting how human hands move in egocentric views is critical for applications like augmented reality and human-robot policy transfer. Recently, several hand trajectory prediction (HTP) methods have been developed to generate future possible hand waypoints, which still suffer from insufficient prediction targets, inherent modality gaps, entangled hand-head motion, and limited validation in downstream tasks. To address these limitations, we present a universal hand motion forecasting framework considering multi-modal input, multi-dimensional and multi-target prediction patterns, and multi-task affordances for downstream applications. We harmonize multiple modalities by vision-language fusion, global context incorporation, and task-aware text embedding injection, to forecast hand waypoints in both 2D and 3D spaces. A novel dual-branch diffusion is proposed to concurrently predict human head and hand movements, capturing their motion synergy in egocentric vision. By introducing target indicators, the prediction model can forecast the specific joint waypoints of the wrist or the fingers, besides the widely studied hand center points. In addition, we enable Uni-Hand to additionally predict hand-object interaction states (contact/separation) to facilitate downstream tasks better. As the first work to incorporate downstream task evaluation in the literature, we build novel benchmarks to assess the real-world applicability of hand motion forecasting algorithms. The experimental results on multiple publicly available datasets and our newly proposed benchmarks demonstrate that Uni-Hand achieves the state-of-the-art performance in multi-dimensional and multi-target hand motion forecasting. Extensive validation in multiple downstream tasks also presents its impressive human-robot policy transfer to enable robotic manipulation, and effective feature enhancement for action anticipation/recognition.

تم تقديم إطار جديد يسمى Uni-Hand للتنبؤ بحركة اليد العالمية في وجهات نظر ذاتية، حيث يتناول التحديات المتعلقة بتوقع حركات اليد البشرية. يدمج هذا الإطار مدخلات متعددة الوسائط ويستخدم نهج انتشار مزدوج الفرع للتنبؤ في الوقت نفسه بحركات اليد والرأس في كل من الفضاءين ثنائي وثلاثي الأبعاد.

Se ha introducido un nuevo marco llamado Uni-Hand para la previsión universal del movimiento de las manos en vistas egocéntricas, abordando los desafíos en la predicción de los movimientos de las manos humanas. Este marco integra entradas multimodales y emplea un enfoque de difusión de doble rama para predecir simultáneamente los movimientos de las manos y la cabeza en espacios 2D y 3D.

Un nouveau cadre nommé Uni-Hand a été introduit pour la prévision universelle des mouvements de la main dans des vues égocentriques, abordant les défis de la prédiction des mouvements de la main humaine. Ce cadre intègre des entrées multimodales et utilise une approche de diffusion à double branche pour prévoir simultanément les mouvements de la main et de la tête dans des espaces 2D et 3D.

A new framework named Uni-Hand has been introduced for universal hand motion forecasting in egocentric views, addressing challenges in predicting human hand movements. This framework integrates multi-modal inputs and employs a dual-branch diffusion approach to simultaneously forecast hand and head movements in both 2D and 3D spaces.

Uni-Hand: Universal Hand Motion Forecasting in Egocentric Views

Betron.io is a blockchain powered prediction and gaming platform where people can forecast market moves, compete in skill based games, and interact with crypto in new ways.

One More Thing in AI – Your Shortcut to AI Mastery

Uni-Hand: Universal Hand Motion Forecasting in Egocentric Views

Was this article worth reading? Share it

Betron.io

One More Thing in AI

Humanize AI

LucidQuery AI

GPTHumanizer

AI Humanizer

Ready to build your own newsroom?