arXiv:2404.10210v5 Announce Type: replace 
Abstract: In recent years, multimodal Graph Convolutional Networks (GCNs) have achieved remarkable performance in skeleton-based action recognition. The reliance on high-energy-consuming continuous floating-point operations inherent in GCN-based methods poses significant challenges for deployment in energy-constrained, battery-powered edge devices. To address these limitations, MK-SGN, a Spiking Graph Convolutional Network with Multimodal Fusion and Knowledge Distillation, is proposed to leverage the energy efficiency of Spiking Neural Networks (SNNs) for skeleton-based action recognition for the first time. By integrating the energy-saving properties of SNNs with the graph representation capabilities of GCNs, MK-SGN achieves significant reductions in energy consumption while maintaining competitive recognition accuracy. Firstly, we formulate a Spiking Multimodal Fusion (SMF) module to effectively fuse multimodal skeleton data represented as spike-form features. Secondly, we propose the Self-Attention Spiking Graph Convolution (SA-SGC) module and the Spiking Temporal Convolution (STC) module, to capture spatial relationships and temporal dynamics of spike-form features. Finally, we propose an integrated knowledge distillation strategy to transfer information from the multimodal GCN to the SGN, incorporating both intermediate-layer distillation and soft-label distillation to enhance the performance of the SGN. MK-SGN exhibits substantial advantages, surpassing state-of-the-art GCN frameworks in energy efficiency and outperforming state-of-the-art SNN frameworks in recognition accuracy. The proposed method achieves a remarkable reduction in energy consumption, exceeding 98\% compared to conventional GCN-based approaches. This research establishes a robust baseline for developing high-performance, energy-efficient SNN-based models for skeleton-based action recognition

MK-SGN هو شبكة جديدة من الشبكات العصبية ذات القمم التي تدمج بين دمج البيانات متعددة الأنماط وتقطير المعرفة من أجل التعرف على الأفعال المستندة إلى الهيكل العظمي. تعالج هذه الطريقة المبتكرة الاستهلاك العالي للطاقة الذي تتطلبه الشبكات التقليدية من خلال الاستفادة من الخصائص الموفرة للطاقة للشبكات العصبية ذات القمم. يحقق MK-SGN تخفيضات كبيرة في استهلاك الطاقة مع الحفاظ على دقة تنافسية في التعرف على الأفعال بناءً على بيانات الهيكل العظمي.

MK-SGN es una nueva red de convolución gráfica de picos que integra fusión multimodal y destilación de conocimiento para el reconocimiento de acciones basado en esqueletos. Este enfoque innovador aborda el alto consumo de energía de las redes de convolución gráfica tradicionales al utilizar las propiedades energéticamente eficientes de las redes neuronales de picos. MK-SGN logra reducciones significativas en el consumo de energía mientras mantiene una precisión competitiva en el reconocimiento de acciones a partir de datos esqueléticos.

MK-SGN est un nouveau réseau de convolution graphique à impulsions qui intègre la fusion multimodale et la distillation des connaissances pour la reconnaissance d'actions basée sur des squelettes. Cette approche innovante répond à la forte consommation d'énergie des réseaux de convolution graphique traditionnels en utilisant les propriétés écoénergétiques des réseaux neuronaux à impulsions. MK-SGN permet de réduire considérablement la consommation d'énergie tout en maintenant une précision compétitive dans la reconnaissance des actions à partir de données squelettiques.

MK-SGN is a newly proposed Spiking Graph Convolutional Network that integrates multimodal fusion and knowledge distillation for skeleton-based action recognition. This innovative approach addresses the high energy consumption of traditional Graph Convolutional Networks by utilizing the energy-efficient properties of Spiking Neural Networks. The MK-SGN achieves significant reductions in energy usage while maintaining competitive accuracy in recognizing actions based on skeletal data.

MK-SGN: A Spiking Graph Convolutional Network with Multimodal Fusion and Knowledge Distillation for Skeleton-based Action Recognition

arXiv:2503.01361v3 Announce Type: replace-cross 
Abstract: Graph neural networks (GNNs) are designed to process data associated with graphs. They are finding an increasing range of applications; however, as with other modern machine learning techniques, their theoretical understanding is limited. GNNs can encounter difficulties in gathering information from nodes that are far apart by iterated aggregation steps. This situation is partly caused by so-called oversmoothing; and overcoming it is one of the practically motivated challenges. We consider the situation where information is aggregated by multiple steps of convolution, leading to graph convolutional networks (GCNs). We analyze the generalization performance of a basic GCN, trained for node classification on data generated by the contextual stochastic block model. We predict its asymptotic performance by deriving the free energy of the problem, using the replica method, in the high-dimensional limit. Calling depth the number of convolutional steps, we show the importance of going to large depth to approach the Bayes-optimality. We detail how the architecture of the GCN has to scale with the depth to avoid oversmoothing. The resulting large depth limit can be close to the Bayes-optimality and leads to a continuous GCN. Technically, we tackle this continuous limit via an approach that resembles dynamical mean-field theory (DMFT) with constraints at the initial and final times. An expansion around large regularization allows us to solve the corresponding equations for the performance of the deep GCN. This promising tool may contribute to the analysis of further deep neural networks.

أجرت دراسة حديثة تحليلًا فيزيائيًا إحصائيًا لشبكات الأعصاب الرسومية (GNNs)، مع التركيز على أدائها في نموذج الكتلة العشوائية السياقية. تسلط الأبحاث الضوء على التحديات التي تواجهها GNNs، وخاصة مشكلة التنعيم الزائد، وتقترح طريقة للتنبؤ بأدائها الأسيمptotic باستخدام طريقة النسخ في حدود الأبعاد العالية.

Un estudio reciente ha realizado un análisis de física estadística de las Redes Neuronales Gráficas (GNN), centrándose en su rendimiento en el modelo de bloque estocástico contextual. La investigación destaca los desafíos que enfrentan las GNN, particularmente el sobreajuste, y propone un método para predecir su rendimiento asintótico utilizando el método de réplicas en límites de alta dimensión.

Une étude récente a réalisé une analyse de la physique statistique des réseaux de neurones graphiques (GNN), en se concentrant sur leur performance dans le modèle de bloc stochastique contextuel. La recherche met en évidence les défis auxquels les GNN sont confrontés, notamment le sur lissage, et propose une méthode pour prédire leur performance asymptotique en utilisant la méthode des répliques dans des limites de haute dimension.

A recent study has conducted a statistical physics analysis of Graph Neural Networks (GNNs), focusing on their performance in the contextual stochastic block model. The research highlights the challenges GNNs face, particularly oversmoothing, and proposes a method to predict their asymptotic performance using the replica method in high-dimensional limits.

MK-SGN: A Spiking Graph Convolutional Network with Multimodal Fusion and Knowledge Distillation for Skeleton-based Action Recognition

Was this article worth reading? Share it

MemeGen AI — Discover what people think of this product.

LucidQuery AI

ClipCutAi