SlimCaching: Edge Caching of Mixture-of-Experts for Distributed Inference

A new model named UniF$^2$ace has been introduced, aimed at addressing challenges in face understanding and generation by unifying these processes into a single framework. This model employs a novel theoretical framework with a Dual Discrete Diffusion (D3Diff) loss, which enhances the precision of facial attribute generation and understanding.

A novel framework called Med-MoE-LoRA has been proposed to enhance the adaptation of Large Language Models (LLMs) for domain-specific applications, particularly in medicine. This framework addresses two significant challenges: the Stability-Plasticity Dilemma and Task Interference, enabling efficient multi-task learning without compromising general knowledge retention.

A recent study titled 'Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models' explores the knowledge acquisition dynamics in Mixture-of-Experts (MoE) architectures compared to dense models, utilizing a new neuron-level attribution metric called Gated-LPI. The research tracks knowledge updates over extensive training steps, revealing significant differences in how these architectures learn.

A recent study on Mixture-of-Experts (MoE) language models reveals that optimal architecture design must consider both total parameters and expert sparsity, rather than relying solely on these factors. The research indicates that increasing the number of experts can negatively impact performance by necessitating reductions in model dimensions to meet memory constraints.