arXiv:2512.10170v1 Announce Type: cross 
Abstract: Automated audio captioning models frequently produce overconfident predictions regardless of semantic accuracy, limiting their reliability in deployment. This deficiency stems from two factors: evaluation metrics based on n-gram overlap that fail to capture semantic correctness, and the absence of calibrated confidence estimation. We present a framework that addresses both limitations by integrating confidence prediction into audio captioning and redefining correctness through semantic similarity. Our approach augments a Whisper-based audio captioning model with a learned confidence prediction head that estimates uncertainty from decoder hidden states. We employ CLAP audio-text embeddings and sentence transformer similarities (FENSE) to define semantic correctness, enabling Expected Calibration Error (ECE) computation that reflects true caption quality rather than surface-level text overlap. Experiments on Clotho v2 demonstrate that confidence-guided beam search with semantic evaluation achieves dramatically improved calibration (CLAP-based ECE of 0.071) compared to greedy decoding baselines (ECE of 0.488), while simultaneously improving caption quality across standard metrics. Our results establish that semantic similarity provides a more meaningful foundation for confidence calibration in audio captioning than traditional n-gram metrics.

تم تقديم إطار عمل جديد للتعليق الصوتي الآلي يدمج توقع الثقة ويعيد تعريف الصحة من خلال التشابه الدلالي. تتناول هذه الطريقة مشكلة التنبؤات المفرطة الثقة في نماذج التعليق الصوتي، التي غالبًا ما تفتقر إلى الدقة الدلالية. من خلال استخدام تمثيلات الصوت والنص CLAP ورأس توقع الثقة المتعلم، يعزز النموذج موثوقية مخرجات التعليق الصوتي.

Se ha introducido un nuevo marco para la subtitulación automática de audio que integra la predicción de confianza y redefine la corrección a través de la similitud semántica. Este enfoque aborda el problema de las predicciones excesivamente confiadas en los modelos de subtitulación de audio, que a menudo carecen de precisión semántica. Al emplear embeddings de audio-texto CLAP y una cabeza de predicción de confianza aprendida, el modelo mejora la fiabilidad de las salidas de subtitulación de audio.

Un nouveau cadre a été introduit pour la légende audio automatisée, intégrant la prédiction de confiance et redéfinissant la justesse par la similarité sémantique. Cette approche aborde le problème des prédictions trop confiantes dans les modèles de légende audio, qui manquent souvent de précision sémantique. En utilisant des embeddings audio-texte CLAP et une tête de prédiction de confiance apprise, le modèle améliore la fiabilité des résultats de légende audio.

A new framework has been introduced for automated audio captioning that integrates confidence prediction and redefines correctness through semantic similarity. This approach addresses the issue of overconfident predictions in audio captioning models, which often lack semantic accuracy. By employing CLAP audio-text embeddings and a learned confidence prediction head, the model enhances the reliability of audio captioning outputs.

Semantic-Aware Confidence Calibration for Automated Audio Captioning

Was this article worth reading? Share it

ShareSpeak

SoundWise.ai

ComfyUI

Dubsmart LLC

Superwhisper

LangWatch

Ready to build your own newsroom?