arXiv:2505.00275v3 Announce Type: replace 
Abstract: Chronic diseases, including diabetes, hypertension, asthma, HIV-AIDS, epilepsy, and tuberculosis, necessitate rigorous adherence to medication to avert disease progression, manage symptoms, and decrease mortality rates. Adherence is frequently undermined by factors including patient behavior, caregiver support, elevated medical costs, and insufficient healthcare infrastructure. We propose AdCare-VLM, a specialized LLaVA-based multimodal large vision language model (LVLM) by introducing a unified visual latent space with pre-alignment to facilitate visual question answering (VQA) concerning medication adherence through patient videos. We employ a private dataset comprising 806 custom-annotated tuberculosis (TB) medication monitoring videos, which have been labeled by clinical experts, to fine-tune the model for adherence pattern detection. We present LLM-TB-VQA, a detailed medical adherence VQA dataset that encompasses positive, negative, and ambiguous adherence cases. Our method identifies correlations between visual features, such as the clear visibility of the patient's face, medication, water intake, and the act of ingestion, and their associated medical concepts in captions. This facilitates the integration of aligned visual-linguistic representations and improves multimodal interactions. Experimental results indicate that our method surpasses parameter-efficient fine-tuning (PEFT) enabled VLM models, such as LLaVA-V1.5 and Chat-UniVi, with absolute improvements ranging from 3.1% to 3.54% across pre-trained, regular, and low-rank adaptation (LoRA) configurations. Comprehensive ablation studies and attention map visualizations substantiate our approach, enhancing interpretability.

AdCare-VLM هو نموذج جديد متعدد الوسائط للغة البصرية مصمم لتحسين الالتزام بالعلاج للأمراض المزمنة مثل السكري وارتفاع ضغط الدم والسل. من خلال استخدام مساحة كامنة بصرية موحدة، يهدف إلى تحسين الإجابة على الأسئلة البصرية المتعلقة بمقاطع الفيديو الخاصة بالمرضى. يتم ضبط النموذج باستخدام مجموعة بيانات تتكون من 806 مقطع فيديو تم وضع علامات عليها لمراقبة الأدوية الخاصة بالسل، مما يسمح بتحسين الكشف عن أنماط الالتزام.

AdCare-VLM es un nuevo modelo multimodal de lenguaje visual diseñado para mejorar la adherencia a la medicación en enfermedades crónicas como la diabetes, la hipertensión y la tuberculosis. Al utilizar un espacio latente visual unificado, busca mejorar la respuesta a preguntas visuales relacionadas con videos de pacientes. El modelo se ajusta utilizando un conjunto de datos de 806 videos anotados sobre la supervisión de la medicación para la tuberculosis, lo que permite una mejor detección de patrones de adherencia.

AdCare-VLM est un nouveau modèle multimodal de langage visuel conçu pour améliorer l'adhésion aux médicaments pour les maladies chroniques telles que le diabète, l'hypertension et la tuberculose. En utilisant un espace latent visuel unifié, il vise à améliorer le questionnement visuel lié aux vidéos des patients. Le modèle est affiné à l'aide d'un ensemble de données de 806 vidéos annotées de surveillance des médicaments contre la tuberculose, permettant une meilleure détection des schémas d'adhésion.

AdCare-VLM is a new multimodal large vision language model designed to enhance medication adherence for chronic diseases such as diabetes, hypertension, and tuberculosis. By utilizing a unified visual latent space, it aims to improve visual question answering related to patient videos. The model is fine-tuned using a dataset of 806 annotated tuberculosis medication monitoring videos, enabling better detection of adherence patterns.

AdCare-VLM: Towards a Unified and Pre-aligned Latent Representation for Healthcare Video Understanding

arXiv:2511.15435v1 Announce Type: new 
Abstract: Advanced multimodal Retrieval-Augmented Generation (MRAG) techniques have been widely applied to enhance the capabilities of Large Multimodal Models (LMMs), but they also bring along novel safety issues. Existing adversarial research has revealed the vulnerability of MRAG systems to knowledge poisoning attacks, which fool the retriever into recalling injected poisoned contents. However, our work considers a different setting: visual attack of MRAG by solely adding imperceptible perturbations at the image inputs of users, without manipulating any other components. This is challenging due to the robustness of fine-tuned retrievers and large-scale generators, and the effect of visual perturbation may be further weakened by propagation through the RAG chain. We propose a novel Hierarchical Visual Attack that misaligns and disrupts the two inputs (the multimodal query and the augmented knowledge) of MRAG's generator to confuse its generation. We further design a hierarchical two-stage strategy to obtain misaligned augmented knowledge. We disrupt the image input of the retriever to make it recall irrelevant knowledge from the original database, by optimizing the perturbation which first breaks the cross-modal alignment and then disrupts the multimodal semantic alignment. We conduct extensive experiments on two widely-used MRAG datasets: OK-VQA and InfoSeek. We use CLIP-based retrievers and two LMMs BLIP-2 and LLaVA as generators. Results demonstrate the effectiveness of our visual attack on MRAG through the significant decrease in both retrieval and generation performance.

تناقش الورقة المعنونة 'HV-Attack: Hierarchical Visual Attack for Multimodal Retrieval Augmented Generation' طريقة جديدة لاستغلال الثغرات في أنظمة الجيل المعزز من خلال الاسترجاع متعدد الوسائط (MRAG). تبرز كيف يمكن أن تؤدي الاضطرابات غير الملحوظة في المدخلات الصورية إلى عدم توافق وتعطيل عملية الجيل، مما يثير مخاوف أمنية كبيرة بالنسبة للنماذج متعددة الوسائط الكبيرة (LMM). تتناول هذه الدراسة تحدي متانة أنظمة MRAG ضد مثل هذه الهجمات البصرية.

El artículo titulado 'HV-Attack: Hierarchical Visual Attack for Multimodal Retrieval Augmented Generation' discute un nuevo método para explotar vulnerabilidades en los sistemas de Generación Aumentada por Recuperación Multimodal (MRAG). Destaca cómo las perturbaciones imperceptibles en las imágenes pueden desalinear y perturbar el proceso de generación, planteando preocupaciones significativas de seguridad para los Modelos Multimodales Grandes (LMM). Esta investigación aborda el desafío de la robustez de los sistemas MRAG frente a tales ataques visuales.

L'article intitulé 'HV-Attack: Hierarchical Visual Attack for Multimodal Retrieval Augmented Generation' aborde une nouvelle méthode pour exploiter les vulnérabilités des systèmes de génération augmentée par récupération multimodale (MRAG). Il met en évidence comment des perturbations imperceptibles des images peuvent désaligner et perturber le processus de génération, soulevant des préoccupations de sécurité significatives pour les grands modèles multimodaux (LMM). Cette recherche traite du défi de la robustesse des systèmes MRAG face à de telles attaques visuelles.

The paper titled 'HV-Attack: Hierarchical Visual Attack for Multimodal Retrieval Augmented Generation' discusses a new method to exploit vulnerabilities in multimodal Retrieval-Augmented Generation (MRAG) systems. It highlights how imperceptible perturbations to image inputs can misalign and disrupt the generation process, posing significant safety concerns for Large Multimodal Models (LMMs). This research addresses the challenge of robustness in MRAG systems against such visual attacks.

AdCare-VLM: Towards a Unified and Pre-aligned Latent Representation for Healthcare Video Understanding

Was this article worth reading? Share it