RouteScan: A Non-Intrusive Approach to Auditing MoE LLMs Safety via Expert Routing Telemetry
The introduction of RouteScan presents a non-intrusive framework for auditing the safety of Mixture-of-Experts (MoE) large language models (LLMs) by leveraging GPU-level expert routing telemetry, addressing the critical need for safety verification in real-world applications.
WPN Brief
- What Happened
The introduction of RouteScan presents a non-intrusive framework for auditing the safety of Mixture-of-Experts (MoE) large language models (LLMs) by leveraging GPU-level expert routing telemetry, addressing the critical need for safety verification in real-world applications.
- Why It Matters
This development is significant as it balances the imperative of ensuring LLM safety against the challenges of user privacy, allowing for effective monitoring without compromising sensitive user data.
- The Bigger Picture
The emergence of RouteScan highlights ongoing concerns regarding the safety and efficiency of MoE architectures, as researchers continue to explore various methodologies to enhance model performance while mitigating risks associated with adversarial inputs and routing imbalances.
Related Reports
More coverage on this story
10 reports across the wire
RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism
The introduction of RotMoLE, a novel framework enhancing the Mixture of Low-Rank Experts (MoE-LoRA) through a rotational gating mechanism, aims to improve the adaptability of Large Language Models (LLMs) in complex scenarios requiring specialized knowledge. This advancement addresses limitations in conventional gating mechanisms that typically apply scalar reweighting to selected experts, thereby enhancing representation and generalization capabilities.
CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
The introduction of CP-MoE, a consistency-preserving Mixture-of-Experts framework, aims to tackle catastrophic forgetting in continual learning for large language models (LLMs) and vision-language models (VLMs). This framework utilizes a transient expert to capture task-specific updates while guiding their integration into stable experts, thereby enhancing knowledge transfer across tasks.
Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts
A new framework called kNN-MoE has been introduced to enhance the routing capabilities of Mixture-of-Experts (MoE) architectures by utilizing a memory of past expert assignments, allowing for more adaptive and efficient token dispatching. This approach addresses the limitations of traditional routers that are often frozen after initial training, which can lead to poor performance under changing data distributions.
RepetitionCurse: Measuring and Understanding Router Imbalance in Mixture-of-Experts LLMs under DoS Stress
arXiv:2512.23995v2 Announce Type: replace-cross Abstract: Mixture-of-Experts architectures have become the standard for scaling large language models due to their superior parameter efficiency. To accommodate the growing number of experts in practice, modern inference systems commonly adopt expert parallelism to distribute experts across devices. However, the absence of explicit load balancing constraints during inference allows adversarial inputs to trigger severe routing concentration. We demonstrate that out-of-distribution prompts can manipulate the routing strategy such that all tokens are consistently routed to the same set of top-$k$ experts, which creates computational bottlenecks on certain devices while forcing others to idle. This converts an efficiency mechanism into a denial-of-service attack vector, leading to violations of service-level agreements for time to first token. We propose RepetitionCurse, a low-cost black-box strategy to exploit this vulnerability. By identifying a universal flaw in MoE router behavior, RepetitionCurse constructs adversarial prompts using simple repetitive token patterns in a model-agnostic manner. On widely deployed MoE models like Mixtral-8x7B, our method increases end-to-end inference latency by 3.063x, degrading service availability significantly.
TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Acceleration
A new framework named TEAM has been proposed to enhance the efficiency of Mixture-of-Experts (MoE) diffusion language models (dLLMs) by reducing the number of activated experts during decoding while increasing the number of accepted tokens. This addresses the inefficiencies observed in current MoE architectures, where many experts are activated but only a few tokens are ultimately utilized, leading to increased inference overhead.
Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
A new framework named Mix-MoE has been proposed to enhance multilingual machine translation (MT) capabilities of large language models (LLMs) by utilizing a mixed Mixture-of-Experts (MoE) architecture. This framework operates in two stages: post-pretraining on monolingual and parallel corpora, effectively separating experts into Language Model Experts and Machine Translation Experts to minimize parameter interference.
Sparse Mixture-of-Experts Routing in Visual Diffusion Transformers:Diagnosis, Boundary Calibration and Evolutionary Roadmap from Routing Collapse to Selective Deadlock
A recent study has diagnosed the training failure modes of Token-Choice sparse Mixture-of-Experts (MoE) in video Diffusion Transformers, identifying five distinct failure modes that impact model performance. The research highlights issues such as global soft saturation and selective deadlock, which hinder the effectiveness of routing mechanisms in these advanced architectures.
Reason--Imagine--Act: Closed-Loop LLM Decision Making with World Models for Autonomous Driving
arXiv:2605.24004v1 Announce Type: cross Abstract: Large language models (LLMs) are promising for autonomous driving, but semantics-only decision policies can yield physically unsafe behavior in dynamic traffic. Existing methods either perform online language reasoning without explicit dynamics verification or use world models mainly in offline pipelines, leaving a gap between semantic intent and physical feasibility at decision time. We propose Reason--Imagine--Act (RIA), a closed-loop framework that couples an LLM reasoner with an action-conditioned world model for online safety verification. At each step, the LLM proposes an action template and candidate sub-actions, the world model performs short-horizon rollouts, and a safety scorer selects the safest executable action with feedback to the next reasoning step. Under a unified CARLA point-goal protocol (1000 episodes), RIA achieves 80.05% route completion, 51.10% arrival rate, and 0.20% collision rate. Under the same closed-loop interface, RIA consistently outperforms training-free baselines, including CARLA TM and MADA, on core closed-loop metrics. For reproducibility, code is available at https://github.com/pku-smart-city/source_code/tree/main/RIA.
The Evaluation Game: Beyond Static LLM Benchmarking
A new study introduces a game-theoretic framework to analyze the interaction between evaluators and trainers in large language models (LLMs), focusing on robustness fine-tuning against jailbreaks. This framework employs group actions to represent data augmentation and explores various regimes based on the trainer's generalization range.
HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
The introduction of Hot-Experts Layer-level Low-Rank Adaptation (HELLoRA) marks a significant advancement in the fine-tuning of Mixture-of-Experts (MoE) models, focusing on attaching Low-Rank Adaptation (LoRA) modules to frequently activated experts, thereby enhancing efficiency and performance.