World PulseNow
Newsletter

A Multimodal Recaptioning Framework to Account for Perceptual Diversity Across Languages in Vision-Language Modeling

arXiv — cs.CV•Wednesday, November 12, 2025 at 5:00:00 AM
PositiveArtificial Intelligence
The development of a multimodal recaptioning framework marks a significant step in addressing the perceptual bias inherent in modern vision-language models (VLMs), which often rely on English translations for training. This bias can lead to a limited understanding of how different cultures and languages describe objects. By incorporating native speaker data and employing multimodal LLM reasoning, the framework enhances the accuracy of image captioning in languages such as German and Japanese. The results demonstrate a notable improvement in text-image retrieval, with mean recall increasing by 3.5 and a 4.4 enhancement in distinguishing native descriptions from translation errors. This framework not only improves the performance of VLMs but also offers insights into cross-dataset and cross-language generalization, paving the way for more culturally aware AI systems that can better serve diverse populations.
— via World Pulse Now AI Editorial System
Read Original

Was this article worth reading? Share it

Recommended Readings
Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generation
arXiv — cs.CV5 hours ago
Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generation
PositiveArtificial Intelligence
The paper introduces VLM3D, a novel framework that utilizes vision-language models (VLMs) to enhance text-to-3D generation. It addresses two major limitations in current models: the lack of fine-grained semantic alignment and inadequate 3D spatial understanding. VLM3D employs a dual-query critic signal to evaluate both semantic fidelity and geometric coherence, significantly improving the generation process. The framework demonstrates its effectiveness across different paradigms, marking a step forward in 3D generation technology.
Read full article
via arXiv — cs.CV
GMAT: Grounded Multi-Agent Clinical Description Generation for Text Encoder in Vision-Language MIL for Whole Slide Image Classification
arXiv — cs.CV5 hours ago
GMAT: Grounded Multi-Agent Clinical Description Generation for Text Encoder in Vision-Language MIL for Whole Slide Image Classification
PositiveArtificial Intelligence
The article presents a new framework called GMAT, which enhances Multiple Instance Learning (MIL) for whole slide image (WSI) classification. By integrating vision-language models (VLMs), GMAT aims to improve the generation of clinical descriptions that are more expressive and medically specific. This addresses limitations in existing methods that rely on large language models (LLMs) for generating descriptions, which often lack domain grounding and detailed medical specificity, thus improving alignment with visual features.
Read full article
via arXiv — cs.CV
Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models
arXiv — cs.LGa day ago
Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models
PositiveArtificial Intelligence
The paper discusses the challenges of test-time prompt tuning for vision-language models, highlighting the issue of prompt optimization bias that can lead to suboptimal performance in downstream tasks. It identifies two main causes: the model's focus on entropy minimization, which may overlook prediction accuracy, and data misalignment between visual and textual modalities. To address these issues, the authors propose a new method called Doubly Debiased Test-Time Prompt Tuning, aimed at improving model performance in zero-shot settings.
Read full article
via arXiv — cs.LG
Preserving Cross-Modal Consistency for CLIP-based Class-Incremental Learning
arXiv — cs.CV2 days ago
Preserving Cross-Modal Consistency for CLIP-based Class-Incremental Learning
PositiveArtificial Intelligence
The paper titled 'Preserving Cross-Modal Consistency for CLIP-based Class-Incremental Learning' addresses the challenges of class-incremental learning (CIL) in vision-language models like CLIP. It introduces a two-stage framework called DMC, which separates the adaptation of the vision encoder from the optimization of textual soft prompts. This approach aims to mitigate classifier bias and maintain cross-modal alignment, enhancing the model's ability to learn new categories without forgetting previously acquired knowledge.
Read full article
via arXiv — cs.CV
FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning
arXiv — cs.CV2 days ago
FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning
PositiveArtificial Intelligence
FastDriveVLA is a novel framework designed for efficient end-to-end autonomous driving through a reconstruction-based visual token pruning method. This approach addresses the high computational costs associated with long visual tokens in Vision-Language-Action (VLA) models. By focusing on retaining visual tokens that contain essential foreground information, FastDriveVLA aims to enhance decision-making in driving scenarios, marking a significant advancement in the application of VLA models in autonomous systems.
Read full article
via arXiv — cs.CV
NP-LoRA: Null Space Projection Unifies Subject and Style in LoRA Fusion
arXiv — cs.CV2 days ago
NP-LoRA: Null Space Projection Unifies Subject and Style in LoRA Fusion
PositiveArtificial Intelligence
The article introduces NP-LoRA, a novel framework for Low-Rank Adaptation (LoRA) fusion that addresses the issue of interference in existing methods. Traditional weight-based merging often leads to one LoRA dominating another, resulting in degraded fidelity. NP-LoRA utilizes a projection-based approach to maintain subspace separation, thereby enhancing the quality of fusion by preventing structural interference among principal directions.
Read full article
via arXiv — cs.CV
CLIPPan: Adapting CLIP as A Supervisor for Unsupervised Pansharpening
arXiv — cs.CV2 days ago
CLIPPan: Adapting CLIP as A Supervisor for Unsupervised Pansharpening
PositiveArtificial Intelligence
The article presents CLIPPan, an unsupervised pansharpening framework that utilizes CLIP, a visual-language model, as a supervisor. This approach addresses the challenges faced by supervised pansharpening methods, particularly the domain adaptation issues arising from the disparity between simulated low-resolution training data and real-world high-resolution scenarios. The framework is designed to improve the understanding of the pansharpening process and enhance the model's ability to recognize various image types, ultimately setting a new state of the art in unsupervised full-resolution pans…
Read full article
via arXiv — cs.CV
Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMs
arXiv — cs.CV2 days ago
Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMs
PositiveArtificial Intelligence
The paper titled 'Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMs' discusses the advancements in Vision-Language Models (VLMs) aimed at enhancing personalization. It highlights the challenges posed by the lack of user-provided positive samples and the poor quality of negative samples. To address these issues, the authors introduce the Concept-as-Tree (CaT) framework, which generates diverse positive and negative samples, thus improving VLM performance in personalization tasks.
Read full article
via arXiv — cs.CV