CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysis

Researchers have found that large language models (LLMs) make decisions about networking and friendship in ways that closely resemble human behavior, both in synthetic simulations and real-world contexts. This suggests that LLMs can replicate social decision-making processes similar to those of people.

A new architecture called Mixture-of-Head attention (MoH) has been proposed to enhance the efficiency of the multi-head attention mechanism, a key component of the Transformer model. This innovation allows tokens to selectively utilize attention heads, improving inference efficiency while maintaining or exceeding previous accuracy levels. MoH replaces the standard summation in multi-head attention with a weighted summation, introducing flexibility and unlocking additional performance potential.

InnoGym has been introduced as the first benchmark and framework aimed at systematically evaluating the innovation potential of AI agents. This initiative focuses on two key metrics: performance gain and novelty, assessing not just the correctness of solutions but also the originality of approaches across 18 tasks from real-world engineering and scientific domains.

A new formulation for reinforcement learning (RL) with large language models (LLMs) has been proposed, emphasizing the optimization of true sequence-level rewards through a surrogate token-level objective in policy gradient methods like REINFORCE. The study highlights the importance of minimizing training-inference discrepancies and policy staleness to enhance the validity of this surrogate, supported by extensive experiments with a 30B Mixture-of-Experts model.

A new quantization method called Four Over Six (4/6) has been introduced to enhance the NVFP4 quantization algorithm, which is crucial for large language models (LLMs). This method evaluates two potential scale factors for each block of values, addressing issues of performance degradation during inference and divergence during training that arise from quantization errors in floating-point formats.

A recent study has demonstrated a method for predicting the performance of black-box language models (LLMs) by utilizing follow-up queries to assess their outputs. This approach allows researchers to train reliable predictors based on the probabilities of responses, achieving accuracy that can surpass traditional white-box models that analyze internal mechanisms.

Recent advancements in Diffusion Mixture-of-Experts (MoE) models have highlighted the importance of architectural configurations over routing mechanisms. A systematic study has identified key factors such as expert modules and attention encodings that significantly enhance the performance of these models, suggesting that tuning these configurations can yield better results than routing innovations alone.

A new framework called CTRL has been developed to teach large language models (LLMs) to critique and refine their outputs through reinforcement learning. This approach allows critic models to generate feedback that enhances the performance of generator models without human intervention, leading to improved pass rates and reduced errors in code generation tasks.