ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization

The paper titled 'Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Models' introduces a method to enhance the efficiency of Mixture-of-Experts (MoE) Large Language Models (LLMs). The authors propose a pre-attention expert prediction technique that improves accuracy and reduces computational overhead by utilizing activations before the attention block. This approach aims to optimize expert prefetching, achieving about a 15% improvement in accuracy over existing methods.

The article introduces Autoregressive Representation Alignment (ARRA), a novel training framework designed to enhance text-to-image generation in autoregressive large language models (LLMs) without altering their architecture. ARRA achieves this by aligning the hidden states of LLMs with visual representations from external models through a global visual alignment loss and a hybrid token. Experimental results demonstrate that ARRA significantly reduces the Fréchet Inception Distance (FID) for models like LlamaGen, indicating improved coherence in generated images.

The paper titled 'Enhanced Structured Lasso Pruning with Class-wise Information' discusses advancements in neural network pruning methods. Traditional pruning techniques often overlook class-wise information, leading to potential loss of statistical data. This study introduces two new pruning schemes, sparse graph-structured lasso pruning with Information Bottleneck (sGLP-IB) and sparse tree-guided lasso pruning with Information Bottleneck (sTLP-IB), aimed at preserving statistical information while reducing model complexity.

Unified Heterogeneous Knowledge Distillation (UHKD) is a proposed framework that enhances knowledge distillation (KD) by utilizing intermediate features in the frequency domain. This approach addresses the limitations of traditional KD methods, which are primarily designed for homogeneous models and struggle in heterogeneous environments. UHKD aims to improve model compression while maintaining accuracy, making it a significant advancement in the field of artificial intelligence.

The paper titled 'NTSFormer: A Self-Teaching Graph Transformer for Multimodal Isolated Cold-Start Node Classification' addresses the challenges of classifying isolated cold-start nodes in multimodal graphs, which often lack edges and modalities. The proposed Neighbor-to-Self Graph Transformer (NTSFormer) employs a self-teaching paradigm to enhance model capacity by using a cold-start attention mask for dual predictions—one based on the node's own features and another guided by a teacher model. This approach aims to improve classification accuracy in scenarios where traditional methods fall sho…

RiverScope is a newly developed high-resolution dataset aimed at improving the monitoring of rivers and surface water dynamics, which are crucial for understanding Earth's climate system. The dataset includes 1,145 high-resolution images covering 2,577 square kilometers, with expert-labeled river and surface water masks. This initiative addresses the challenges of monitoring narrow or sediment-rich rivers that are often inadequately represented in low-resolution satellite data.