VLM as Strategist: Adaptive Generation of Safety-critical Testing Scenarios via Guided Diffusion

The Large Shapes and Textures dataset (LAS&T) has been introduced to enhance the capabilities of Large Vision-Language Models (LVLMs) in recognizing and representing shapes and textures across various contexts. This dataset, created through unsupervised extraction from natural images, serves as a benchmark for evaluating the performance of leading models like CLIP and DINO in shape recognition tasks.

A recent study introduces Function-word De-Attention (FDA) as a method to enhance the robustness of Vision-Language Models (VLMs) against cross-modal adversarial attacks by reducing the influence of function words. The FDA technique differentiates between original and function-word cross-attention, leading to improved alignment and robustness in VLMs. Comprehensive experiments demonstrate significant reductions in attack success rates with minimal performance drops across various models and tasks.

OpenSubject has been introduced as a large-scale video-derived dataset comprising 2.5 million samples and 4.35 million images, aimed at improving subject-driven image generation and manipulation. This dataset employs a four-stage pipeline that utilizes cross-frame identity priors to enhance the accuracy of generated images in complex scenes with multiple subjects.

A recent study has conducted a visual comparison between Vision Foundation Models (VFMs) and Vision Language Models (VLMs) for 3D pose estimation, particularly in hand object grasping scenarios. The research highlights the strengths of CLIP in semantic understanding and DINOv2 in providing dense geometric features, demonstrating their complementary roles in enhancing 6D object pose estimation.

A recent study highlights the challenges faced by Vision Language Models (VLMs) in detecting AI-generated images (AIGI), revealing that fine-tuning on high-level semantic supervision improves performance, while low-level pixel-artifact supervision leads to poor results. This misalignment between task and model capabilities is a core issue affecting detection accuracy.

The VOST-SGG framework has been introduced as a one-stage spatio-temporal scene graph generation model that leverages vision-language models (VLMs) to enhance the understanding of object relationships in video frames. This approach addresses limitations in existing models, such as uninformed query initialization and reliance on unimodal features for predicate classification.