Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships

Pre-trained vision-language models are increasingly susceptible to adversarial attacks, with existing defense strategies primarily targeting image classification. This limitation has prompted researchers to explore more comprehensive solutions. The introduction of Multimodal Adversarial Training (MAT) marks a significant advancement, as it incorporates adversarial perturbations in both image and text modalities during training. This approach not only outperforms traditional unimodal defenses but also addresses the inherent vulnerabilities of multimodal tasks. A key insight from the study is the importance of leveraging one-to-many relationships, where a single image can correspond to multiple textual descriptions and vice versa. By conducting a thorough analysis of augmentation techniques, the researchers found that well-aligned and diverse augmented image-text pairs can enhance the robustness of these models. This work not only pioneers defense strategies against multimodal attacks bu…