Compositional Semantics for Open Vocabulary Spatio-semantic Representations
A recent study introduces latent compositional semantic embeddings (z*) as a novel knowledge representation for queryable spatio-semantic memories, enhancing the capabilities of vision-language models (VLMs) to reason beyond immediate percepts. The research demonstrates that z* can be derived from visual data and descriptions through iterative gradient descent, with experimental validation across multiple embedding spaces like CLIP and SBERT.
WPN Brief
- What Happened
A recent study introduces latent compositional semantic embeddings (z*) as a novel knowledge representation for queryable spatio-semantic memories, enhancing the capabilities of vision-language models (VLMs) to reason beyond immediate percepts. The research demonstrates that z* can be derived from visual data and descriptions through iterative gradient descent, with experimental validation across multiple embedding spaces like CLIP and SBERT.
- Why It Matters
This development is significant as it addresses the limitations of current VLMs, enabling them to perform complex reasoning tasks that require understanding beyond what is visually present. The findings suggest that z* can effectively represent multiple semantics, potentially improving the performance of applications reliant on VLMs.
- The Bigger Picture
The introduction of z* aligns with ongoing efforts to enhance multimodal understanding in AI, as seen in recent benchmarks evaluating reasoning capabilities in multimodal large language models (MLLMs). This highlights a growing focus on improving AI's ability to integrate and reason across different modalities, which is crucial for applications in areas such as disaster management and embodied navigation.
Related Reports
More coverage on this story
10 reports across the wire
Is Inference Mediated by Distinct Semantic Structures in LLMs? A Mechanistic Interpretation
A recent study published on arXiv investigates the mechanisms of inference in large language models (LLMs), specifically examining whether distinct semantic structures mediate inference processes. The research employs controlled premise-hypothesis pairs to assess how semantic transformations affect predictions, revealing that transformer representations carry operation-level information with significant accuracy.
TimeSpot: Benchmarking Geo-Temporal Understanding in Vision-Language Models in Real-World Settings
TimeSpot has been introduced as a benchmark for evaluating geo-temporal understanding in vision-language models (VLMs), focusing on the ability to infer location and time from visual inputs. This benchmark includes 1,455 images from 80 countries and requires structured predictions of temporal and geographic attributes.
Uncovering Autoregressive LLM Knowledge of Thematic Fit in Event Representation
Recent research has demonstrated that autoregressive large language models (LLMs) possess varying degrees of knowledge regarding the thematic fit of event arguments, achieving new benchmarks in this area. The study involved manipulating input contexts and reasoning strategies to evaluate how different models respond to thematic fit tasks.
VEN-VL: A Visual Ensemble MoE Framework for Effective and Efficient Multi-Modal Understanding
arXiv:2605.25952v1 Announce Type: new Abstract: Despite the remarkable progress achieved by recent efficient methods in accelerating multimodal understanding, they still suffer from noticeable performance degradation. Their emphasis on the high compression ratio of a single visual clue and reliance on the heuristic pruning strategy with coarse attention alignment incurs a bottleneck on the information capacity and density of visual tokens. Addressing this limitation, we propose VEN-VL, a visual ensemble MoE framework for effective and efficient perception following the enrich then compact principle. Specifically, we first enrich the information capacity by unifying the visual representations of different perspectives, and then progressively compact it with adaptive routers in specialized visual experts to enhance the information density. Furthermore, we incorporate the reconstruction ability of vanilla structure via explicit visual supervision, facilitating crucial information preservation. Experimental results demonstrate our superiority in complex visual tasks with few information-condensed tokens, which effectively bridges the gap between performance and efficiency.
Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning
arXiv:2605.25364v1 Announce Type: new Abstract: Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.
Hide to Guide: Learning via Semantic Masking
arXiv:2605.25198v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a powerful paradigm for improving language models on reasoning-intensive tasks, but its effectiveness is often limited by exploration. For example, models often fail on hard problems, leaving little useful reward signal. External expert traces offer a natural source of guidance, yet they may also expose reward-relevant content along the critical path to the verifier target, such as final answers, intermediate values, executable implementations, or answer-related entities. This content can create an unintended reward hacking channel, allowing the policy to obtain reward by copying the trace rather than learning the underlying reasoning or agentic behavior. Existing guided-RL methods reduce this risk by using partial trajectories, but they mainly control how much expert information is shown heuristically rather than which parts should be hidden. To this end, we propose Semantic Masked Expert Policy Optimization (SMEPO), a fine-grained semantic masking strategy for expert-guided RLVR. Instead of truncating traces coarsely or revealing them unchanged, SMEPO masks reward-relevant semantic spans along the critical path while preserving the expert's decomposition, plan, and procedural structure. This turns hard problems from reasoning from scratch into a fill-in-the-blank process: the policy can follow the expert's problem-solving route, but must still reconstruct the missing values, code, or entities by itself. SMEPO is simple to apply and requires no changes to the reward function or RL objective. Across diverse domains, including math, code, and agentic search, SMEPO improves accuracy by up to 3.2 points over GRPO and reduces training time by up to 4.2x. The code is available at https://github.com/mit-han-lab/SMEPO.
Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs
Recent research has highlighted a significant performance gap in multimodal large language models (MLLMs) when processing text as images compared to traditional text tokens. This study evaluated seven MLLMs across various benchmarks and found that the modality gap is sensitive to rendering choices, with natural document images showing smaller gaps. The primary cause identified is that image input alone suppresses reasoning effort, leading to shorter outputs that skip detailed computation.
$M^3-Verse$: A "Spot the Difference" Challenge for Large Multimodal Models
arXiv:2512.18735v2 Announce Type: replace Abstract: Modern Large Multimodal Models (LMMs) have demonstrated extraordinary ability in static image and single-state spatial-temporal understanding. However, their capacity to comprehend the dynamic changes of objects within a shared spatial context between two distinct video observations, remains largely unexplored. This ability to reason about transformations within a consistent environment is particularly crucial for advancements in the field of spatial intelligence. In this paper, we introduce $M^3-Verse$, a Multi-Modal, Multi-State, Multi-Dimensional benchmark, to formally evaluate this capability. It is built upon paired videos that provide multi-perspective observations of an indoor scene before and after a state change. The benchmark contains a total of 270 scenes and 2,932 questions, which are categorized into over 50 subtasks that probe 4 core capabilities. We evaluate 16 state-of-the-art LMMs and observe their limitations in tracking state transitions. To address these challenges, we further propose a simple yet effective baseline that achieves significant performance improvements in multi-state perception. $M^3-Verse$ thus provides a challenging new testbed to catalyze the development of next-generation models with a more holistic understanding of our dynamic visual world. You can get the construction pipeline from https://github.com/Wal-K-aWay/M3-Verse_pipeline and full benchmark data from https://www.modelscope.cn/datasets/WalKaWay/M3-Verse.
Neural Router: Semantic Content Matching for Agentic AI
The recent study on the Neural Router presents a novel approach to semantic content matching for agentic AI, utilizing large language models (LLMs) as a semantic-matching engine within a content-based publish/subscribe broker. This framework aims to address the challenges posed by vocabulary and modality gaps that hinder traditional keyword and embedding filters.
MAIL++: Multi-Modal Bi-directional Agent Layer for Vision-Language Models
arXiv:2605.25479v1 Announce Type: new Abstract: Adapting large vision-language models (VLMs) such as CLIP to downstream tasks remains challenging, as full fine-tuning is computationally prohibitive and prone to overfitting in low-data regimes. Parameter-efficient fine-tuning (PEFT) alleviates these issues with lightweight prompt- or adapter-based modules, and cross-modal coupling has proven especially effective by strengthening interactions between vision and language. However, existing coupling mechanisms predominantly rely on external auxiliary modules, leading to indirect, coarse-grained interactions that are structurally decoupled from the original VLM and thus limit representational expressiveness. In this paper, we propose Multi-Modal Interactive Agent Layer (MAIL), a PEFT paradigm that embeds cross-modal coupling directly into the intrinsic computation modules of VLMs. MAIL freezes the backbone and inserts lightweight agent layers after core modules, such as LayerNorm, to approximate the parameter updates induced by full fine-tuning. To couple visual and textual streams at this level, we introduce a bottleneck-based text-to-image bridge that jointly optimizes paired agent layers across modalities, coordinating the adaptation of corresponding computation modules. We further present MAIL++, which enables bidirectional cross-modal exchange through a meta agent layer, a meta-text bridge, and a meta-image bridge. At inference time, all agent layers are re-parameterized into the frozen backbone, preserving the original computational efficiency. Extensive experiments on few-shot image classification and few-shot universal cross-domain retrieval demonstrate that MAIL and MAIL++ consistently outperform state-of-the-art PEFT methods.