Spiking Neural Networks Need High Frequency Information

The study on MPD-SGR explores the surrogate gradient method's potential to enhance deep spiking neural networks (SNNs) while addressing their vulnerabilities to adversarial attacks. It highlights the importance of gradient magnitude, which indicates the model's sensitivity to input changes. The research reveals that by reducing the proportion of membrane potentials within the gradient-available range of the surrogate gradient function, the robustness of SNNs can be significantly improved.

The paper titled 'Bias-Restrained Prefix Representation Finetuning for Mathematical Reasoning' introduces a new method called Bias-REstrained Prefix Representation FineTuning (BREP ReFT). This approach aims to enhance the mathematical reasoning capabilities of models by addressing the limitations of existing Representation finetuning (ReFT) methods, which struggle with mathematical tasks. The study demonstrates that BREP ReFT outperforms both standard ReFT and weight-based Parameter-Efficient finetuning (PEFT) methods through extensive experiments.

The study investigates the ability of transformer models to predict long steps in the Collatz sequence, a complex arithmetic function that maps odd integers to their successors. The accuracy of the models varies significantly depending on the base used for encoding, achieving up to 99.7% accuracy for bases 24 and 32, while dropping to 37% and 25% for bases 11 and 3. Despite these variations, all models exhibit a common learning pattern, accurately predicting inputs with similar residuals modulo 2^p.

The article discusses CHNNet, an innovative artificial neural network that incorporates intra-layer connections among hidden neurons, contrasting with traditional hierarchical architectures that limit direct neuron interactions within the same layer. This new design aims to enhance information flow and integration, potentially leading to faster convergence rates compared to conventional feedforward neural networks. Experimental results support the theoretical predictions regarding the model's performance.

Vision-Language Models (VLMs) are emerging models that integrate visual content with natural language. Current methods typically fuse data either early in the encoding process or late through pooled embeddings. This paper introduces a lightweight fusion module utilizing cross-only, bidirectional attention layers to align hidden states from both modalities, enhancing understanding while keeping encoders non-causal. The proposed method aims to improve the performance of VLMs by leveraging the inherent structure of visual and textual data.

Higher-order Neural Additive Models (HONAMs) have been introduced as an advancement over Neural Additive Models (NAMs), which are known for their predictive performance and interpretability. HONAMs address the limitation of NAMs by effectively capturing feature interactions of arbitrary orders, enhancing predictive accuracy while maintaining interpretability, crucial for high-stakes applications. The source code for HONAM is publicly available on GitHub.

Spiking Neural Networks (SNNs) are gaining popularity due to their energy efficiency, but they face challenges in effective training. Recent advancements have introduced knowledge distillation (KD) techniques, utilizing pre-trained artificial neural networks (ANNs) as teachers for SNNs. This process typically aligns features and predictions from both networks, but often overlooks their architectural differences. To address this, two new KD strategies, Saliency-scaled Activation Map Distillation (SAMD) and Noise-smoothed Logits Distillation (NLD), have been proposed to enhance training effectiv…

The paper titled 'StochEP: Stochastic Equilibrium Propagation for Spiking Convergent Recurrent Neural Networks' introduces a new framework for training Spiking Neural Networks (SNNs) using Stochastic Equilibrium Propagation (EP). This method aims to enhance training stability and scalability by integrating probabilistic spiking neurons, addressing limitations of traditional Backpropagation Through Time (BPTT) and deterministic EP approaches. The proposed framework shows promise in narrowing performance gaps in vision benchmarks.