MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique

The launch of MM-CRITIC represents a pivotal moment in the evaluation of Large Multimodal Models (LMMs), focusing on their ability to critique and self-improve. This benchmark covers eight main task types and includes over 500 tasks, providing a comprehensive framework for assessing LMMs' performance across multiple dimensions. With a sample size of 4,471, the evaluation integrates expert-informed ground answers into its scoring rubrics, ensuring reliable assessments. Extensive experiments have validated the effectiveness of MM-CRITIC, revealing insights into the correlation between response quality and critique, as well as the varying difficulty of critique across evaluation dimensions. As AI continues to evolve, tools like MM-CRITIC are essential for enhancing the capabilities of AI assistants, ensuring they can provide accurate and reliable support in diverse applications.