A Multimodal Recaptioning Framework to Account for Perceptual Diversity Across Languages in Vision-Language Modeling

The development of a multimodal recaptioning framework marks a significant step in addressing the perceptual bias inherent in modern vision-language models (VLMs), which often rely on English translations for training. This bias can lead to a limited understanding of how different cultures and languages describe objects. By incorporating native speaker data and employing multimodal LLM reasoning, the framework enhances the accuracy of image captioning in languages such as German and Japanese. The results demonstrate a notable improvement in text-image retrieval, with mean recall increasing by 3.5 and a 4.4 enhancement in distinguishing native descriptions from translation errors. This framework not only improves the performance of VLMs but also offers insights into cross-dataset and cross-language generalization, paving the way for more culturally aware AI systems that can better serve diverse populations.