Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models

A new model named UniF$^2$ace has been introduced, aimed at addressing challenges in face understanding and generation by unifying these processes into a single framework. This model employs a novel theoretical framework with a Dual Discrete Diffusion (D3Diff) loss, which enhances the precision of facial attribute generation and understanding.

A novel framework called Med-MoE-LoRA has been proposed to enhance the adaptation of Large Language Models (LLMs) for domain-specific applications, particularly in medicine. This framework addresses two significant challenges: the Stability-Plasticity Dilemma and Task Interference, enabling efficient multi-task learning without compromising general knowledge retention.

A recent study on Mixture-of-Experts (MoE) language models reveals that optimal architecture design must consider both total parameters and expert sparsity, rather than relying solely on these factors. The research indicates that increasing the number of experts can negatively impact performance by necessitating reductions in model dimensions to meet memory constraints.