arXiv:2503.14960v3 Announce Type: replace 
Abstract: Skeleton-based Human Action Recognition (HAR) is a vital technology in robotics and human-robot interaction. However, most existing methods concentrate primarily on full-body movements and often overlook subtle hand motions that are critical for distinguishing fine-grained actions. Recent work leverages a unified graph representation that combines body, hand, and foot keypoints to capture detailed body dynamics. Yet, these models often blur fine hand details due to the disparity between body and hand action characteristics and the loss of subtle features during the spatial-pooling. In this paper, we propose BHaRNet (Body-Hand action Recognition Network), a novel framework that augments a typical body-expert model with a hand-expert model. Our model jointly trains both streams with an ensemble loss that fosters cooperative specialization, functioning in a manner reminiscent of a Mixture-of-Experts (MoE). Moreover, cross-attention is employed via an expertized branch method and a pooling-attention module to enable feature-level interactions and selectively fuse complementary information. Inspired by MMNet, we also demonstrate the applicability of our approach to multi-modal tasks by leveraging RGB information, where body features guide RGB learning to capture richer contextual cues. Experiments on large-scale benchmarks (NTU RGB+D 60, NTU RGB+D 120, PKU-MMD, and Northwestern-UCLA) demonstrate that BHaRNet achieves SOTA accuracies -- improving from 86.4\% to 93.0\% in hand-intensive actions -- while maintaining fewer GFLOPs and parameters than the relevant unified methods.

يعد التعرف على حركة الإنسان القائم على الهيكل العظمي (HAR) تقنية حيوية في الروبوتات والتفاعل بين الإنسان والروبوت. ومع ذلك، تركز معظم الطرق الحالية بشكل أساسي على الحركات الكاملة للجسم وغالبًا ما تتجاهل الحركات اليدوية الدقيقة التي تعتبر حاسمة لتمييز الأفعال الدقيقة. يقدم هذا البحث BHaRNet، وهو إطار عمل يعزز نموذج خبير الجسم بنموذج خبير اليد.

El reconocimiento de acciones humanas basado en esqueletos (HAR) es crucial para la robótica y la interacción humano-robot. Los métodos tradicionales a menudo se centran en los movimientos de todo el cuerpo, descuidando los sutiles movimientos de las manos que son esenciales para distinguir acciones finas. Este artículo presenta BHaRNet, un marco que integra modelos expertos de cuerpo y mano, mejorando el reconocimiento a través de la especialización cooperativa y mecanismos de atención cruzada.

La reconnaissance d'action humaine basée sur le squelette (HAR) est essentielle pour la robotique et l'interaction homme-robot. Les méthodes traditionnelles se concentrent souvent sur les mouvements du corps entier, négligeant les mouvements subtils des mains qui sont cruciaux pour différencier les actions fines. Cet article présente BHaRNet, un cadre qui intègre des modèles experts du corps et de la main, améliorant la reconnaissance grâce à une spécialisation coopérative et des mécanismes d'attention croisée.

Skeleton-based Human Action Recognition (HAR) is crucial for robotics and human-robot interaction. Traditional methods often focus on full-body movements, neglecting subtle hand motions essential for fine-grained action differentiation. This paper introduces BHaRNet, a framework that integrates body and hand expert models, enhancing recognition through cooperative specialization and cross-attention mechanisms.

Body-Hand Modality Expertized Networks with Cross-attention for Fine-grained Skeleton Action Recognition

arXiv:2511.22433v2 Announce Type: replace 
Abstract: Recent advances in skeleton-based action recognition increasingly leverage semantic priors from Large Language Models (LLMs) to enrich skeletal representations. However, the LLM is typically queried in isolation from the recognition model and receives no performance feedback. As a result, it often fails to deliver the targeted discriminative cues critical to distinguish similar actions. To overcome these limitations, we propose SkeletonAgent, a novel framework that bridges the recognition model and the LLM through two cooperative agents, i.e., Questioner and Selector. Specifically, the Questioner identifies the most frequently confused classes and supplies them to the LLM as context for more targeted guidance. Conversely, the Selector parses the LLM's response to extract precise joint-level constraints and feeds them back to the recognizer, enabling finer-grained cross-modal alignment. Comprehensive evaluations on five benchmarks, including NTU RGB+D, NTU RGB+D 120, Kinetics-Skeleton, FineGYM, and UAV-Human, demonstrate that SkeletonAgent consistently outperforms state-of-the-art benchmark methods. The code is available at https://github.com/firework8/SkeletonAgent.

تم تقديم إطار SkeletonAgent لتحسين التعرف على الأفعال المعتمد على الهياكل العظمية من خلال دمج نماذج اللغة الكبيرة (LLMs) مع نموذج التعرف عبر وكيلين تعاونيين، وهما السائل والمحدد. تهدف هذه الطريقة المبتكرة إلى تحسين دقة التمييز بين الأفعال المتشابهة من خلال تقديم إرشادات مستهدفة وتغذية راجعة بين LLM ونموذج التعرف.

Se ha presentado el marco SkeletonAgent para mejorar el reconocimiento de acciones basado en esqueletos, integrando Modelos de Lenguaje de Gran Tamaño (LLMs) con un modelo de reconocimiento a través de dos agentes cooperativos, el Cuestionador y el Selector. Este enfoque innovador busca mejorar la precisión en la distinción de acciones similares al proporcionar orientación y retroalimentación específicas entre el LLM y el modelo de reconocimiento.

Le cadre SkeletonAgent a été introduit pour améliorer la reconnaissance d'actions basée sur des squelettes en intégrant des modèles de langage de grande taille (LLMs) avec un modèle de reconnaissance à travers deux agents coopératifs, le Questionneur et le Sélecteur. Cette approche innovante vise à améliorer la précision de la distinction entre des actions similaires en fournissant des conseils ciblés et des retours d'information entre le LLM et le modèle de reconnaissance.

The SkeletonAgent framework has been introduced to enhance skeleton-based action recognition by integrating Large Language Models (LLMs) with a recognition model through two cooperative agents, the Questioner and Selector. This innovative approach aims to improve the accuracy of distinguishing similar actions by providing targeted guidance and feedback between the LLM and the recognition model.

Body-Hand Modality Expertized Networks with Cross-attention for Fine-grained Skeleton Action Recognition

Was this article worth reading? Share it

Octofy

GPTHumanizer

Fakeface