Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints

A new model named UniF$^2$ace has been introduced, aimed at addressing challenges in face understanding and generation by unifying these processes into a single framework. This model employs a novel theoretical framework with a Dual Discrete Diffusion (D3Diff) loss, which enhances the precision of facial attribute generation and understanding.

A novel framework called Med-MoE-LoRA has been proposed to enhance the adaptation of Large Language Models (LLMs) for domain-specific applications, particularly in medicine. This framework addresses two significant challenges: the Stability-Plasticity Dilemma and Task Interference, enabling efficient multi-task learning without compromising general knowledge retention.

A recent study titled 'Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models' explores the knowledge acquisition dynamics in Mixture-of-Experts (MoE) architectures compared to dense models, utilizing a new neuron-level attribution metric called Gated-LPI. The research tracks knowledge updates over extensive training steps, revealing significant differences in how these architectures learn.