arXiv:2505.00275v3 Announce Type: replace 
Abstract: Chronic diseases, including diabetes, hypertension, asthma, HIV-AIDS, epilepsy, and tuberculosis, necessitate rigorous adherence to medication to avert disease progression, manage symptoms, and decrease mortality rates. Adherence is frequently undermined by factors including patient behavior, caregiver support, elevated medical costs, and insufficient healthcare infrastructure. We propose AdCare-VLM, a specialized LLaVA-based multimodal large vision language model (LVLM) by introducing a unified visual latent space with pre-alignment to facilitate visual question answering (VQA) concerning medication adherence through patient videos. We employ a private dataset comprising 806 custom-annotated tuberculosis (TB) medication monitoring videos, which have been labeled by clinical experts, to fine-tune the model for adherence pattern detection. We present LLM-TB-VQA, a detailed medical adherence VQA dataset that encompasses positive, negative, and ambiguous adherence cases. Our method identifies correlations between visual features, such as the clear visibility of the patient's face, medication, water intake, and the act of ingestion, and their associated medical concepts in captions. This facilitates the integration of aligned visual-linguistic representations and improves multimodal interactions. Experimental results indicate that our method surpasses parameter-efficient fine-tuning (PEFT) enabled VLM models, such as LLaVA-V1.5 and Chat-UniVi, with absolute improvements ranging from 3.1% to 3.54% across pre-trained, regular, and low-rank adaptation (LoRA) configurations. Comprehensive ablation studies and attention map visualizations substantiate our approach, enhancing interpretability.

AdCare-VLM هو نموذج جديد متعدد الوسائط للغة البصرية مصمم لتحسين الالتزام بالعلاج للأمراض المزمنة مثل السكري وارتفاع ضغط الدم والسل. من خلال استخدام مساحة كامنة بصرية موحدة، يهدف إلى تحسين الإجابة على الأسئلة البصرية المتعلقة بمقاطع الفيديو الخاصة بالمرضى. يتم ضبط النموذج باستخدام مجموعة بيانات تتكون من 806 مقطع فيديو تم وضع علامات عليها لمراقبة الأدوية الخاصة بالسل، مما يسمح بتحسين الكشف عن أنماط الالتزام.

AdCare-VLM es un nuevo modelo multimodal de lenguaje visual diseñado para mejorar la adherencia a la medicación en enfermedades crónicas como la diabetes, la hipertensión y la tuberculosis. Al utilizar un espacio latente visual unificado, busca mejorar la respuesta a preguntas visuales relacionadas con videos de pacientes. El modelo se ajusta utilizando un conjunto de datos de 806 videos anotados sobre la supervisión de la medicación para la tuberculosis, lo que permite una mejor detección de patrones de adherencia.

AdCare-VLM est un nouveau modèle multimodal de langage visuel conçu pour améliorer l'adhésion aux médicaments pour les maladies chroniques telles que le diabète, l'hypertension et la tuberculose. En utilisant un espace latent visuel unifié, il vise à améliorer le questionnement visuel lié aux vidéos des patients. Le modèle est affiné à l'aide d'un ensemble de données de 806 vidéos annotées de surveillance des médicaments contre la tuberculose, permettant une meilleure détection des schémas d'adhésion.

AdCare-VLM is a new multimodal large vision language model designed to enhance medication adherence for chronic diseases such as diabetes, hypertension, and tuberculosis. By utilizing a unified visual latent space, it aims to improve visual question answering related to patient videos. The model is fine-tuned using a dataset of 806 annotated tuberculosis medication monitoring videos, enabling better detection of adherence patterns.

AdCare-VLM: Towards a Unified and Pre-aligned Latent Representation for Healthcare Video Understanding

arXiv:2406.09838v4 Announce Type: replace 
Abstract: Meteorological heatmaps play a vital role in deciphering extreme weather phenomena, yet their inherent complexities marked by irregular contours, unstructured patterns, and complex color variations present unique analytical hurdles for state-of-the-art Vision-Language Models (VLMs). Current state-of-the-art models like GPT-4o, Qwen-VL, and LLaVA 1.6 struggle with tasks such as precise color identification and spatial localization, resulting in inaccurate or incomplete interpretations. To address these challenges, we introduce Sparse Position and Outline Tracking (SPOT), a novel algorithm specifically designed to process irregularly shaped colored regions in visual data. SPOT identifies and localizes these regions by extracting their spatial coordinates, enabling structured representations of irregular shapes. Building on SPOT, we construct ClimateIQA, a novel meteorological visual question answering (VQA) dataset, comprising 26,280 high-resolution heatmaps and 762,120 instruction samples for wind gust, total precipitation, wind chill index and heat index analysis. ClimateIQA enhances VLM training by incorporating spatial cues, geographic metadata, and reanalysis data, improving model accuracy in interpreting and describing extreme weather features. Furthermore, we develop Climate-Zoo, a suite of fine-tuned VLMs based on SPOT-empowered ClimateIQA, which significantly outperforms existing models in meteorological heatmap tasks.

تم تقديم مجموعة بيانات جديدة تُدعى ClimateIQA لتعزيز قدرات نماذج اللغة البصرية (VLM) في تحليل الشذوذات الجوية. تشمل هذه المجموعة 26,280 صورة عالية الجودة، وتهدف إلى معالجة التحديات التي تواجه النماذج الحالية مثل GPT-4o وQwen-VL في تفسير خرائط الحرارة الجوية المعقدة التي تتميز بأشكال غير منتظمة وتباينات لونية.

Se ha introducido un nuevo conjunto de datos llamado ClimateIQA para mejorar las capacidades de los Modelos de Lenguaje Visual (VLM) en el análisis de anomalías meteorológicas. Este conjunto de datos, que incluye 26,280 imágenes de alta calidad, busca abordar los desafíos que enfrentan modelos existentes como GPT-4o y Qwen-VL al interpretar mapas de calor meteorológicos complejos caracterizados por formas irregulares y variaciones de color.

Un nouveau jeu de données nommé ClimateIQA a été introduit pour améliorer les capacités des modèles de langage visuel (VLM) dans l'analyse des anomalies météorologiques. Ce jeu de données, qui comprend 26 280 images de haute qualité, vise à relever les défis rencontrés par les modèles existants comme GPT-4o et Qwen-VL dans l'interprétation des cartes thermiques météorologiques complexes, caractérisées par des formes irrégulières et des variations de couleur.

A new dataset named ClimateIQA has been introduced to enhance the capabilities of Vision-Language Models (VLMs) in analyzing meteorological anomalies. This dataset, which includes 26,280 high-quality images, aims to address the challenges faced by existing models like GPT-4o and Qwen-VL in interpreting complex meteorological heatmaps characterized by irregular shapes and color variations.

ClimateIQA: A New Dataset and Benchmark to Advance Vision-Language Models in Meteorology Anomalies Analysis

arXiv:2601.03184v2 Announce Type: replace 
Abstract: We present a theoretical analysis of decentralization of autoregressive generation. We define the Decentralized Discrete Flow Matching objective, by expressing probability generating velocity as a linear combination of expert flows. We also conduct experiments demonstrating the equivalence between decentralized and centralized training settings for multimodal language models across diverse set of benchmarks. Specifically, we compare two distinct paradigms: LLaVA and InternVL 2.5-1B, which uses a fixed CLIP vision encoder and performs full-parameter fine-tuning (ViT+MLP+LLM) during the instruction tuning stage.

تم تقديم تحليل نظري للامركزية في التوليد الذاتي، حيث تم تقديم هدف مطابقة التدفق المنفصل اللامركزي، الذي يعبر عن سرعة توليد الاحتمالات كمزيج خطي من تدفقات الخبراء. تُظهر التجارب التكافؤ بين إعدادات التدريب اللامركزية والمركزية لنماذج اللغة متعددة الوسائط، مع مقارنة محددة بين LLaVA وInternVL 2.5-1B.

Se ha presentado un análisis teórico de la descentralización en la generación autorregresiva, introduciendo el objetivo de Coincidencia de Flujo Discreto Descentralizado, que expresa la velocidad de generación de probabilidad como una combinación lineal de flujos de expertos. Los experimentos demuestran la equivalencia entre los entornos de entrenamiento descentralizados y centralizados para modelos de lenguaje multimodal, comparando específicamente LLaVA e InternVL 2.5-1B.

Une analyse théorique de la décentralisation dans la génération autorégressive a été présentée, introduisant l'objectif de Correspondance de Flux Discret Décentralisé, qui exprime la vitesse de génération de probabilité comme une combinaison linéaire de flux d'experts. Des expériences démontrent l'équivalence entre les paramètres d'entraînement décentralisés et centralisés pour les modèles de langage multimodaux, en comparant spécifiquement LLaVA et InternVL 2.5-1B.

A theoretical analysis of decentralization in autoregressive generation has been presented, introducing the Decentralized Discrete Flow Matching objective, which expresses probability generating velocity as a linear combination of expert flows. Experiments demonstrate the equivalence between decentralized and centralized training settings for multimodal language models, specifically comparing LLaVA and InternVL 2.5-1B.

AdCare-VLM: Towards a Unified and Pre-aligned Latent Representation for Healthcare Video Understanding

Was this article worth reading? Share it

LucidQuery AI

Avela Health

Thinkability

Vocablii

Twofold Health

Nudge AI

Ready to build your own newsroom?