Unveiling Super Experts in Mixture-of-Experts Large Language Models

arXiv — cs.CL•Thursday, November 13, 2025 at 5:00:00 AM

The discovery of Super Experts (SEs) in Mixture-of-Experts (MoE) large language models (LLMs) marks a significant advancement in understanding model performance. SEs, characterized by rare but extreme activation outliers, play a pivotal role in the forward inference of MoE LLMs. The study reveals that pruning just three out of 6,144 SEs in the Qwen3-30B-A3B model results in repetitive and uninformative outputs, underscoring their critical function. This model-specific and data-agnostic distribution of SEs remains unaffected by post-training processes, indicating their inherent significance. The analysis further demonstrates that compressing SEs disrupts the systematic outlier mechanism, leading to a collapse of attention sinks, which profoundly impacts the model's overall performance, especially in tasks requiring mathematical reasoning. This research not only sheds light on the intricate workings of LLMs but also emphasizes the need to preserve these Super Experts for optimal model fu…

— via World Pulse Now AI Editorial System

Read Original

Was this article worth reading? Share it

Recommended Readings

arXiv — cs.LG7 hours ago

DeepBlip: Estimating Conditional Average Treatment Effects Over Time

PositiveArtificial Intelligence

DeepBlip is a novel neural framework designed to estimate conditional average treatment effects over time using structural nested mean models (SNMMs). This approach allows for the decomposition of treatment sequences into localized, time-specific 'blip effects', enhancing interpretability and enabling efficient evaluation of treatment policies. DeepBlip integrates sequential neural networks like LSTMs and transformers, addressing the limitations of existing methods by allowing simultaneous learning of all blip functions.

Read full article

via arXiv — cs.LG

arXiv — cs.LG7 hours ago

Bayes optimal learning of attention-indexed models

PositiveArtificial Intelligence

The paper introduces the attention-indexed model (AIM), a framework for analyzing learning in deep attention layers. AIM captures the emergence of token-level outputs from bilinear interactions over high-dimensional embeddings. It allows full-width key and query matrices, aligning with practical transformers. The study derives predictions for Bayes-optimal generalization error and identifies phase transitions based on sample complexity, model width, and sequence length, proposing a message passing algorithm and demonstrating optimal performance via gradient descent.

Read full article

via arXiv — cs.LG

arXiv — cs.LGa day ago

CLAReSNet: When Convolution Meets Latent Attention for Hyperspectral Image Classification

PositiveArtificial Intelligence

CLAReSNet, a new hybrid architecture for hyperspectral image classification, integrates multi-scale convolutional extraction with transformer-style attention through an adaptive latent bottleneck. This model addresses challenges such as high spectral dimensionality, complex spectral-spatial correlations, and limited training samples with severe class imbalance. By combining convolutional networks and transformers, CLAReSNet aims to enhance classification accuracy and efficiency in hyperspectral imaging applications.

Read full article

via arXiv — cs.LG

arXiv — cs.CV2 days ago

RiverScope: High-Resolution River Masking Dataset

PositiveArtificial Intelligence

RiverScope is a newly developed high-resolution dataset aimed at improving the monitoring of rivers and surface water dynamics, which are crucial for understanding Earth's climate system. The dataset includes 1,145 high-resolution images covering 2,577 square kilometers, with expert-labeled river and surface water masks. This initiative addresses the challenges of monitoring narrow or sediment-rich rivers that are often inadequately represented in low-resolution satellite data.

Read full article

via arXiv — cs.CV

$$\pi$-Attention: Periodic Sparse Transformers for Efficient Long-Context Modeling$

arXiv — cs.CL2 days ago

$\pi$-Attention: Periodic Sparse Transformers for Efficient Long-Context Modeling

PositiveArtificial Intelligence

Read full article

via arXiv — cs.CL

arXiv — cs.LG2 days ago

Multistability of Self-Attention Dynamics in Transformers

NeutralArtificial Intelligence

The paper titled 'Multistability of Self-Attention Dynamics in Transformers' explores a continuous-time multiagent model of self-attention mechanisms in transformers. It establishes a connection between self-attention dynamics and a multiagent version of the Oja flow, which computes the principal eigenvector of a matrix related to the value matrix in transformers. The study classifies the equilibria of the single-head self-attention system into four categories: consensus, bipartite consensus, clustering, and polygonal equilibria, noting that multiple stable equilibria can coexist.

Read full article

via arXiv — cs.LG

arXiv — cs.LG2 days ago

Transformers know more than they can tell -- Learning the Collatz sequence

NeutralArtificial Intelligence

The study investigates the ability of transformer models to predict long steps in the Collatz sequence, a complex arithmetic function that maps odd integers to their successors. The accuracy of the models varies significantly depending on the base used for encoding, achieving up to 99.7% accuracy for bases 24 and 32, while dropping to 37% and 25% for bases 11 and 3. Despite these variations, all models exhibit a common learning pattern, accurately predicting inputs with similar residuals modulo 2^p.

Read full article

via arXiv — cs.LG