PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions

A recent study investigates bias in generative AI text annotations, replicating manual annotations from Boukes (2024) using various Generative Large Language Models (GLLMs) including Llama3.1, Llama3.3, GPT4o, and Qwen2.5. The findings indicate that while GLLMs achieve adequate F1 scores, they exhibit systematic bias, aligning more closely with each other than with manual annotations, which leads to different downstream results.

A novel approach called SATGround has been introduced to enhance visual grounding in remote sensing through a structured localization mechanism that fine-tunes a pretrained vision-language model (VLM) on diverse instruction-following tasks. This method significantly improves the model's ability to localize objects in complex satellite imagery, achieving a 24.8% relative improvement over previous methods in visual grounding benchmarks.

The Embodied Tree of Thoughts (EToT) framework has been introduced as a significant advancement in robot manipulation planning, utilizing a physics-based interactive digital twin to enhance the prediction of future environmental states and the reasoning of actions prior to execution. This approach aims to overcome limitations found in existing video-generation models, which often lack physical grounding and consistency in long-horizon constraints.

A new approach to procedural mistake detection (PMD) has been introduced, focusing on classifying task execution success through egocentric video analysis. This method emphasizes generating visual self-dialog rationales to enhance decision-making transparency, leveraging advanced vision-and-language models (VLMs) and establishing baseline metrics for coherence in generated rationales.

CORE-3D introduces a novel approach to 3D scene understanding by utilizing context-aware open-vocabulary retrieval through embeddings, enhancing the accuracy of object-level masks in complex environments. This method leverages SemanticSAM and a refined CLIP encoding strategy to improve 3D semantic segmentation, addressing limitations of previous models that produced fragmented masks and inaccurate semantic assignments.

A new framework named LaFG has been introduced for fine-grained image retrieval, which utilizes large language models (LLMs) and vision-language models (VLMs) to convert class names into detailed attribute-level descriptions. This approach aims to enhance the modeling of comparability among cross-category details, addressing limitations of existing methods that rely on sparse one-hot labels.

A new framework called AerialVP has been introduced to enhance image perception in UAVs by improving task prompts used in Vision-Language Models (VLMs). This framework addresses challenges such as target confusion and scale variations that arise from the complex nature of UAV imagery, which traditional VLMs struggle to interpret effectively.