MoH: Multi-Head Attention as Mixture-of-Head Attention

Researchers have found that large language models (LLMs) make decisions about networking and friendship in ways that closely resemble human behavior, both in synthetic simulations and real-world contexts. This suggests that LLMs can replicate social decision-making processes similar to those of people.

InnoGym has been introduced as the first benchmark and framework aimed at systematically evaluating the innovation potential of AI agents. This initiative focuses on two key metrics: performance gain and novelty, assessing not just the correctness of solutions but also the originality of approaches across 18 tasks from real-world engineering and scientific domains.

A recent study introduced Semantically Equivalent and Coherent Attacks (SECA) aimed at eliciting hallucinations in Large Language Models (LLMs) through realistic prompt modifications that maintain semantic coherence. This approach addresses the limitations of previous adversarial attacks that often resulted in unrealistic prompts, thereby enhancing the understanding of how LLMs can produce hallucinations in practical applications.

A recent study has demonstrated a method for predicting the performance of black-box language models (LLMs) by utilizing follow-up queries to assess their outputs. This approach allows researchers to train reliable predictors based on the probabilities of responses, achieving accuracy that can surpass traditional white-box models that analyze internal mechanisms.

A new quantization method called Four Over Six (4/6) has been introduced to enhance the NVFP4 quantization algorithm, which is crucial for large language models (LLMs). This method evaluates two potential scale factors for each block of values, addressing issues of performance degradation during inference and divergence during training that arise from quantization errors in floating-point formats.

Recent advancements in Diffusion Mixture-of-Experts (MoE) models have highlighted the importance of architectural configurations over routing mechanisms. A systematic study has identified key factors such as expert modules and attention encodings that significantly enhance the performance of these models, suggesting that tuning these configurations can yield better results than routing innovations alone.

A new framework called CTRL has been developed to teach large language models (LLMs) to critique and refine their outputs through reinforcement learning. This approach allows critic models to generate feedback that enhances the performance of generator models without human intervention, leading to improved pass rates and reduced errors in code generation tasks.

A new framework named CORE has been proposed for trajectory representation learning (TRL), which aims to enhance the encoding of raw trajectory data into low-dimensional embeddings by integrating context-aware route choice semantics. This approach addresses the limitations of existing TRL methods that treat trajectories as static sequences, thereby enriching the semantic representation of urban mobility patterns.