MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping

The paper introduces MOON, a generative Multimodal Large Language Model (MLLM) aimed at enhancing product representation learning in e-commerce. It addresses challenges such as the lack of multimodal modeling modules and background noise in product images. The proposed model seeks to improve the alignment between multiple images and texts associated with products, marking a significant advancement in e-commerce product understanding.

Multimodal Large Language Models (MLLMs) have shown remarkable capabilities in tasks such as OCR and VQA, but hallucination remains a significant challenge. This paper is the first to explore verb hallucination in MLLMs, revealing that many state-of-the-art models exhibit severe issues with verb concepts. The study evaluates existing methods aimed at reducing hallucinations related to object concepts and assesses their effectiveness on verb hallucinations.

The paper presents ANTS, an innovative method for enhancing Out-of-Distribution (OOD) detection by utilizing Adaptive Negative Textual Space. By leveraging multimodal large language models (MLLMs), the approach generates expressive negative sentences that accurately characterize OOD distributions. This method addresses the limitations of existing techniques, particularly in near-OOD detection, by caching images likely to be OOD samples and prompting MLLMs for detailed descriptions.

A comprehensive study on visual token redundancy in discrete diffusion-based multimodal large language models (dMLLMs) has been conducted, revealing significant computational overhead during inference due to full-sequence attention. The research highlights that visual redundancy primarily occurs in from-scratch dMLLMs when addressing long-answer tasks and examines the impact of visual token pruning on model efficiency and responses.

OmniSparse introduces a training-aware fine-grained sparse attention framework aimed at enhancing the performance of long-video multimodal large language models (MLLMs). Unlike existing methods that focus on inference-time acceleration, OmniSparse operates effectively during both training and inference by dynamically allocating token budgets. This approach includes mechanisms for query selection and key-value selection, addressing the limitations of traditional sparse attention methods.

The paper introduces MoE-SpeQ, a novel inference system designed to address the memory limitations of Mixture-of-Experts (MoE) models during inference. Traditional methods often lead to I/O bottlenecks due to data-dependent expert selection. MoE-SpeQ mitigates this by utilizing a small on-device draft model to predict future expert requirements, allowing for proactive prefetching from host memory. This approach enhances performance by reducing the critical path of execution and improving overall efficiency in MoE applications.

AdaTok introduces an innovative object-level token merging strategy for Adaptive Token compression, aimed at enhancing the efficiency of Multimodal Large Language Models (MLLMs). Traditional patch-level tokenization has resulted in excessive computational and memory demands, leading to misalignments with human cognitive processes. The proposed method significantly reduces token usage to 10% while maintaining nearly 96% of the original model's performance, addressing critical challenges in multimodal understanding and reasoning.

The paper introduces a new Mixture-of-Experts framework for object detection, which utilizes adaptive routing among multiple YOLOv9-T experts. This approach allows for dynamic feature specialization, resulting in improved performance metrics, specifically higher mean Average Precision (mAP) and Average Recall (AR) compared to using a single YOLOv9-T model. The findings suggest significant advancements in the field of object detection.