Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts
A new framework called kNN-MoE has been introduced to enhance the routing capabilities of Mixture-of-Experts (MoE) architectures by utilizing a memory of past expert assignments, allowing for more adaptive and efficient token dispatching. This approach addresses the limitations of traditional routers that are often frozen after initial training, which can lead to poor performance under changing data distributions.
WPN Brief
- What Happened
A new framework called kNN-MoE has been introduced to enhance the routing capabilities of Mixture-of-Experts (MoE) architectures by utilizing a memory of past expert assignments, allowing for more adaptive and efficient token dispatching. This approach addresses the limitations of traditional routers that are often frozen after initial training, which can lead to poor performance under changing data distributions.
- Why It Matters
The kNN-MoE framework is significant as it improves the performance of large language models by enabling them to leverage historical data for better decision-making, thus potentially reducing the need for extensive supervised fine-tuning.
- The Bigger Picture
This development reflects a broader trend in artificial intelligence towards integrating retrieval-augmented methods and knowledge-driven approaches, as seen in other innovations like knowledge graph-driven reasoning and efficient expert pruning techniques, which aim to enhance the efficiency and effectiveness of AI systems.
Related Reports
More coverage on this story
2 reports across the wire
HEAPr: Hessian-based Efficient Atomic Expert Pruning in Output Space
arXiv:2509.22299v3 Announce Type: replace Abstract: Mixture-of-Experts (MoE) architectures in large language models (LLMs) deliver exceptional performance and reduced inference costs compared to dense LLMs. However, their large parameter counts result in prohibitive memory requirements, limiting practical deployment. While existing pruning methods primarily focus on expert-level pruning, this coarse granularity often leads to substantial accuracy degradation. In this work, we introduce HEAPr, a novel pruning algorithm that decomposes experts into smaller, indivisible atomic experts, enabling more precise and flexible atomic expert pruning. To measure the importance of each atomic expert, we leverage second-order information based on principles similar to the Optimal Brain Surgeon theory. To address the computational and storage challenges posed by second-order information, HEAPr exploits the inherent properties of atomic experts to transform the second-order information from expert parameters into that of atomic expert parameters, and further simplifies it to the second-order information of atomic expert outputs. This approach reduces the space complexity from $O(d^4)$, where $d$ is the model's dimensionality, to $O(d^2)$. HEAPr requires only two forward passes and one backward pass on a small calibration set to compute the importance of atomic experts. Extensive experiments on MoE models, including DeepSeek MoE and Qwen MoE family, demonstrate that HEAPr outperforms existing expert-level pruning methods across a wide range of pruning ratios and benchmarks. Specifically, HEAPr achieves nearly lossless compression at pruning ratios of 20% ~ 25% in most models, while also reducing FLOPs nearly by 20%. The code can be found at [https://github.com/LLIKKE/HEAPr](https://github.com/LLIKKE/HEAPr).
Knowledge Graph-Driven Expert-Level Reasoning for Neuroscience
A recent study has explored the potential of knowledge graphs (KGs) to enhance expert-level reasoning in neuroscience by utilizing information from a single authoritative textbook. The research posits that a well-structured KG can enable fine-tuned language models (LMs) to achieve superior reasoning capabilities, surpassing larger models in accuracy while using significantly fewer parameters.