Quality Over Quantity? LLM-Based Curation for a Data-Efficient Audio-Video Foundation Model

The introduction of the Audio-Video Vector Alignment (AVVA) framework marks a significant advancement in the training of multimodal foundational models, addressing the challenges of integrating audio and visual data. By employing Large Language Models for data curation, AVVA not only streamlines the training process but also enhances the accuracy of video-to-audio retrieval tasks. Evaluations on datasets such as AudioCaps, VALOR, and VGGSound reveal that AVVA achieves notable improvements in top-k accuracies compared to the previous DenseAV model, all while utilizing a mere 192 hours of curated training data. This innovative approach underscores the importance of data quality over sheer quantity, as the ablation study indicates that the curation process effectively trades off data quality for improved retrieval accuracies. The implications of this research extend beyond mere technical advancements, suggesting a paradigm shift in how multimodal models can be trained more efficiently and…