Red Teaming Multimodal Language Models: Evaluating Harm Across Prompt Modalities and Models

A new dataset named ClimateIQA has been introduced to enhance the capabilities of Vision-Language Models (VLMs) in analyzing meteorological anomalies. This dataset, which includes 26,280 high-quality images, aims to address the challenges faced by existing models like GPT-4o and Qwen-VL in interpreting complex meteorological heatmaps characterized by irregular shapes and color variations.

The research titled 'LLaVAction' focuses on evaluating and training multi-modal large language models (MLLMs) for action understanding, reformulating the EPIC-KITCHENS-100 dataset into a benchmark for MLLMs. The study reveals that leading MLLMs struggle with recognizing correct actions when faced with difficult distractors, highlighting a gap in their fine-grained action understanding capabilities.

DriveRX has been introduced as a vision-language reasoning model aimed at enhancing cross-task autonomous driving by addressing the limitations of traditional end-to-end models, which struggle with complex scenarios due to a lack of structured reasoning. This model is part of a broader framework called AutoDriveRL, which optimizes four core tasks through a unified training approach.