IFEval-Audio: Benchmarking Instruction-Following Capability in Audio-based Large Language Models

The introduction of the IFEval-Audio dataset marks a significant step in evaluating instruction-following capabilities in audio-based large language models (LLMs). While large language models have shown proficiency in following instructions for text-based tasks, their performance often declines when integrated with non-text modalities like audio. This dataset, consisting of 280 audio-instruction-answer triples across six diverse dimensions—Content, Capitalization, Symbol, List Structure, Length, and Format—aims to benchmark state-of-the-art audio LLMs in this area. The public release of IFEval-Audio is crucial as it fills a notable gap in research, providing a foundation for future studies and advancements in the instruction-following performance of audio LLMs.