The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss

A recent study explores sound symbolism, revealing how Multimodal Large Language Models (MLLMs) interpret auditory information in human languages. The research introduces LEX-ICON, a dataset comprising 8,052 words and 2,930 pseudo-words across four languages, examining MLLMs' phonetic iconicity through phoneme-level attention scores.

A recent study has introduced a method called nlg2choice, aimed at enhancing the capabilities of Multimodal Large Language Models (MLLMs) in Fine-Grained Visual Classification (FGVC). This approach addresses the challenges of evaluating free-form responses in auto-regressive models, particularly in settings with extensive multiple-choice options where traditional methods fall short.

A new paradigm called One-shot video-Clip based Retrieval AuGmentation (OneClip-RAG) has been proposed to enhance the efficiency of Multimodal Large Language Models (MLLMs) in processing long videos, addressing the limitations of existing models that can only handle a limited number of frames due to memory constraints.

MiniGPT-5 has been introduced as a novel interleaved vision-and-language generation model that utilizes generative vokens to enhance the coherence of image-text outputs. This model employs a two-stage training strategy that allows for description-free multimodal generation, significantly improving performance on datasets like MMDialog and VIST.

Recent advancements in Multimodal Large Language Models (MLLMs) have led to the development of See-Control, a framework designed for smartphone interaction with a robotic arm. This framework introduces the Embodied Smartphone Operation (ESO) task, allowing for platform-agnostic smartphone operation through direct physical interaction, bypassing the limitations of the Android Debug Bridge (ADB). See-Control includes an ESO benchmark, an MLLM-based agent, and a dataset of operation episodes.

A novel framework named UniME has been introduced to enhance multimodal representation learning by addressing limitations in existing models like CLIP, particularly in text token truncation and isolated encoding. This two-stage approach utilizes Multimodal Large Language Models (MLLMs) to learn discriminative representations for various tasks, aiming to break the modality barrier in AI applications.

A recent study has highlighted the critical issue of privacy leakage in Multimodal Large Language Models (MLLMs), emphasizing the need for effective recovery of user privacy. The research introduces the SPPE dataset, which simulates various MLLM applications and assesses the quality of privacy recovery through surrogate-driven data restoration. This approach aims to bridge the gap in existing methodologies that focus primarily on obscuring private information without evaluating recovery authenticity.

The introduction of the Visual Reasoning Sequential Attack (VRSA) highlights vulnerabilities in Multimodal Large Language Models (MLLMs), which are increasingly used for their advanced cross-modal capabilities. This method decomposes harmful text into sequential sub-images, allowing MLLMs to externalize harmful intent more effectively.