arXiv:2505.16791v3 Announce Type: replace 
Abstract: Real-world machine learning applications often involve data from multiple modalities that must be integrated effectively to make robust predictions. However, in many practical settings, not all modalities are available for every sample, and acquiring additional modalities can be costly. This raises the question: which samples should be prioritized for additional modality acquisition when resources are limited? While prior work has explored individual-level acquisition strategies and training-time active learning paradigms, test-time and cohort-based acquisition remain underexplored. We introduce Cohort-based Active Modality Acquisition (CAMA), a novel test-time setting to formalize the challenge of selecting which samples should receive additional modalities. We derive acquisition strategies that leverage a combination of generative imputation and discriminative modeling to estimate the expected benefit of acquiring missing modalities based on common evaluation metrics. We also introduce upper-bound heuristics that provide performance ceilings to benchmark acquisition strategies. Experiments on multimodal datasets with up to 15 modalities demonstrate that our proposed imputation-based strategies can more effectively guide the acquisition of additional modalities for selected samples compared with methods relying solely on unimodal information, entropy-based guidance, or random selection. We showcase the real-world relevance and scalability of our method by demonstrating its ability to effectively guide the costly acquisition of proteomics data for disease prediction in a large prospective cohort, the UK Biobank (UKBB). Our work provides an effective approach for optimizing modality acquisition at the cohort level, enabling more effective use of resources in constrained settings.

يقدم إدخال نظام اكتساب النماذج النشطة القائم على المجموعات (CAMA) نهجًا مبتكرًا لتحسين اكتساب البيانات الإضافية في تطبيقات التعلم الآلي، خاصة عندما تكون الموارد محدودة. تتناول هذه الطريقة التحدي المتمثل في اختيار العينات التي يجب أن تتلقى نماذج إضافية لتحسين دقة التنبؤ، من خلال الاستفادة من الاستيفاء التوليدي والنمذجة التمييزية.

La introducción de la Adquisición Activa de Modalidades Basada en Cohortes (CAMA) presenta un enfoque novedoso para optimizar la adquisición de datos adicionales en aplicaciones de aprendizaje automático, especialmente cuando los recursos son limitados. Este método aborda el desafío de seleccionar qué muestras deben recibir modalidades adicionales para mejorar la precisión predictiva, aprovechando la imputación generativa y la modelización discriminativa.

L'introduction de l'Acquisition Active de Modalités Basée sur des Cohortes (CAMA) propose une approche novatrice pour optimiser l'acquisition de données supplémentaires dans les applications d'apprentissage automatique, en particulier lorsque les ressources sont limitées. Cette méthode aborde le défi de la sélection des échantillons devant recevoir des modalités supplémentaires pour améliorer la précision prédictive, en s'appuyant sur l'imputation générative et la modélisation discriminative.

The introduction of Cohort-based Active Modality Acquisition (CAMA) presents a novel approach to optimizing the acquisition of additional data modalities in machine learning applications, particularly when resources are limited. This method addresses the challenge of selecting which samples should receive extra modalities to enhance predictive accuracy, leveraging generative imputation and discriminative modeling.

Cohort-Based Active Modality Acquisition

arXiv:2505.17097v3 Announce Type: replace 
Abstract: Multimodal in-context learning (ICL) is becoming a key capability that allows large vision-language models (LVLMs) to adapt to novel tasks without parameter updates, which expands their usefulness in many real-world applications. However, ICL performance remains unstable even when the in-context demonstrations (ICDs) are well matched, showing that LVLMs still struggle to make full use of the provided context. While existing work mainly focuses on prompt engineering or post-hoc logit calibration, we study the attention mechanisms inside LVLMs to address their inherent limitations. We identify two important weaknesses in their self-attention that hinder effective ICL. To address these weaknesses, we propose \textbf{Context-Aware Modulated Attention} (CAMA), a training-free and plug-and-play method that dynamically adjusts attention logits based on the input in-context sequence. CAMA uses a two-stage modulation process that strengthens attention to semantically important tokens, especially visual ones. Across four LVLMs and seven benchmarks, CAMA consistently outperforms vanilla models and baselines, showing clear effectiveness and generalization. It can also activate the intended benefits of prompt engineering methods and remains robust across different sequence configurations. Therefore, CAMA opens up new directions for improving multimodal reasoning through a deeper understanding of attention dynamics.

اقترحت دراسة حديثة استخدام الانتباه المعدل الواعي بالسياق (CAMA) لتحسين أداء نماذج الرؤية واللغة الكبيرة (LVLMs) في التعلم متعدد الوسائط في السياق (ICL). تتناول هذه الطريقة القيود الجوهرية في آليات الانتباه الذاتي، التي أعاقت نماذج LVLMs من الاستفادة الكاملة من السياق المقدم، حتى مع وجود عروض سياقية متطابقة بشكل جيد.

Un estudio reciente ha propuesto la Atención Modulada Consciente del Contexto (CAMA) para mejorar el rendimiento de los grandes modelos de visión-lenguaje (LVLMs) en el aprendizaje multimodal en contexto (ICL). Este método aborda las limitaciones inherentes en los mecanismos de auto-atención, que han obstaculizado a los LVLMs para utilizar plenamente el contexto proporcionado, incluso con demostraciones en contexto bien emparejadas.

Une étude récente a proposé l'Attention Modulaire Contextuelle (CAMA) pour améliorer les performances des grands modèles de vision-langage (LVLMs) dans l'apprentissage multimodal en contexte (ICL). Cette méthode aborde les limitations inhérentes aux mécanismes d'auto-attention, qui ont empêché les LVLMs d'utiliser pleinement le contexte fourni, même avec des démonstrations en contexte bien assorties.

A recent study has proposed Context-Aware Modulated Attention (CAMA) to enhance the performance of large vision-language models (LVLMs) in multimodal in-context learning (ICL). This method addresses inherent limitations in self-attention mechanisms, which have hindered LVLMs from fully utilizing provided context, even with well-matched in-context demonstrations.

Cohort-Based Active Modality Acquisition

Was this article worth reading? Share it

LucidQuery AI

Https

ClassX