SpeechJudge: Towards Human-Level Judgment for Speech Naturalness

The introduction of SpeechJudge marks a significant advancement in the field of speech synthesis, addressing the critical challenge of aligning generative models with human feedback. This initiative includes the creation of SpeechJudge-Data, a comprehensive dataset featuring 99,000 speech pairs annotated for intelligibility and naturalness, and SpeechJudge-Eval, a benchmark designed to assess speech naturalness. The evaluation of current models, particularly Gemini-2.5-Flash, reveals a concerning gap, with less than 70% agreement with human judgments. This highlights the pressing need for improvement in existing metrics and models, as they struggle to accurately reflect human preferences in speech synthesis. By bridging this gap, SpeechJudge aims to enhance the quality of speech generation, making it more aligned with human perception and ultimately improving user experience in applications relying on speech technology.