LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders

AlignBench has been introduced as a benchmark for evaluating fine-grained image-text alignment using synthetic image-caption pairs, addressing limitations in existing models like CLIP that rely on rule-based perturbations or short captions. This benchmark allows for a more detailed assessment of visual-language models (VLMs) by annotating each sentence for correctness.

CT-GLIP, a new 3D Grounded Language-Image Pretrained model, has been introduced to enhance the alignment of CT scans with radiology reports, addressing limitations in existing methods that rely on global embeddings. This model constructs fine-grained CT-report pairs to improve cross-modal contrastive learning, enabling better identification of organs and abnormalities in a zero-shot manner.

A new Mixture-of-Ranks (MoR) architecture has been proposed for one-step real-world image super-resolution (Real-ISR), integrating sparse Mixture-of-Experts (MoE) to enhance the adaptability of models in reconstructing high-resolution images from degraded samples. This approach utilizes a fine-grained expert partitioning strategy, treating each rank in Low-Rank Adaptation (LoRA) as an independent expert, thereby improving the model's ability to capture heterogeneous characteristics of real-world images.

A new paper introduces VIVAT, a systematic approach designed to mitigate common artifacts in the training of Variational Autoencoders (VAEs), which are crucial for generative computer vision. The study identifies five prevalent artifacts and proposes modifications to improve VAE performance, achieving state-of-the-art results in image reconstruction metrics and enhancing text-to-image generation quality.

A new framework named Prompt-OT has been introduced to enhance the adaptation of vision-language models (VLMs) like CLIP, addressing challenges related to overfitting and zero-shot generalization during fine-tuning. This optimal transport-guided approach preserves the structural consistency of feature distributions between pre-trained and fine-tuned models, ensuring effective prompt learning.