Understanding Hardness of Vision-Language Compositionality from A Token-level Causal Lens

A recent study explores the limitations of Contrastive Language-Image Pre-training (CLIP) in understanding compositional reasoning. While CLIP excels at aligning images and texts, it struggles with complex relationships and attributes, often treating inputs like a simple bag of words. This research highlights the importance of token-level analysis, which could lead to improvements in how AI systems interpret and generate language in relation to visual content. Understanding these challenges is crucial for advancing AI's capabilities in real-world applications.