arXiv:2404.10210v5 Announce Type: replace 
Abstract: In recent years, multimodal Graph Convolutional Networks (GCNs) have achieved remarkable performance in skeleton-based action recognition. The reliance on high-energy-consuming continuous floating-point operations inherent in GCN-based methods poses significant challenges for deployment in energy-constrained, battery-powered edge devices. To address these limitations, MK-SGN, a Spiking Graph Convolutional Network with Multimodal Fusion and Knowledge Distillation, is proposed to leverage the energy efficiency of Spiking Neural Networks (SNNs) for skeleton-based action recognition for the first time. By integrating the energy-saving properties of SNNs with the graph representation capabilities of GCNs, MK-SGN achieves significant reductions in energy consumption while maintaining competitive recognition accuracy. Firstly, we formulate a Spiking Multimodal Fusion (SMF) module to effectively fuse multimodal skeleton data represented as spike-form features. Secondly, we propose the Self-Attention Spiking Graph Convolution (SA-SGC) module and the Spiking Temporal Convolution (STC) module, to capture spatial relationships and temporal dynamics of spike-form features. Finally, we propose an integrated knowledge distillation strategy to transfer information from the multimodal GCN to the SGN, incorporating both intermediate-layer distillation and soft-label distillation to enhance the performance of the SGN. MK-SGN exhibits substantial advantages, surpassing state-of-the-art GCN frameworks in energy efficiency and outperforming state-of-the-art SNN frameworks in recognition accuracy. The proposed method achieves a remarkable reduction in energy consumption, exceeding 98\% compared to conventional GCN-based approaches. This research establishes a robust baseline for developing high-performance, energy-efficient SNN-based models for skeleton-based action recognition

MK-SGN هو شبكة جديدة من الشبكات العصبية ذات القمم التي تدمج بين دمج البيانات متعددة الأنماط وتقطير المعرفة من أجل التعرف على الأفعال المستندة إلى الهيكل العظمي. تعالج هذه الطريقة المبتكرة الاستهلاك العالي للطاقة الذي تتطلبه الشبكات التقليدية من خلال الاستفادة من الخصائص الموفرة للطاقة للشبكات العصبية ذات القمم. يحقق MK-SGN تخفيضات كبيرة في استهلاك الطاقة مع الحفاظ على دقة تنافسية في التعرف على الأفعال بناءً على بيانات الهيكل العظمي.

MK-SGN es una nueva red de convolución gráfica de picos que integra fusión multimodal y destilación de conocimiento para el reconocimiento de acciones basado en esqueletos. Este enfoque innovador aborda el alto consumo de energía de las redes de convolución gráfica tradicionales al utilizar las propiedades energéticamente eficientes de las redes neuronales de picos. MK-SGN logra reducciones significativas en el consumo de energía mientras mantiene una precisión competitiva en el reconocimiento de acciones a partir de datos esqueléticos.

MK-SGN est un nouveau réseau de convolution graphique à impulsions qui intègre la fusion multimodale et la distillation des connaissances pour la reconnaissance d'actions basée sur des squelettes. Cette approche innovante répond à la forte consommation d'énergie des réseaux de convolution graphique traditionnels en utilisant les propriétés écoénergétiques des réseaux neuronaux à impulsions. MK-SGN permet de réduire considérablement la consommation d'énergie tout en maintenant une précision compétitive dans la reconnaissance des actions à partir de données squelettiques.

MK-SGN is a newly proposed Spiking Graph Convolutional Network that integrates multimodal fusion and knowledge distillation for skeleton-based action recognition. This innovative approach addresses the high energy consumption of traditional Graph Convolutional Networks by utilizing the energy-efficient properties of Spiking Neural Networks. The MK-SGN achieves significant reductions in energy usage while maintaining competitive accuracy in recognizing actions based on skeletal data.

MK-SGN: A Spiking Graph Convolutional Network with Multimodal Fusion and Knowledge Distillation for Skeleton-based Action Recognition

Was this article worth reading? Share it

LucidQuery AI

Graphite Note

SVGenius

Golan AI

Gracker AI

sync. labs

Ready to build your own newsroom?