How to Evaluate Speech Translation with Source-Aware Neural MT Metrics

The recent study on source-aware metrics for speech translation (ST) addresses a critical gap in the evaluation of ST systems, particularly in scenarios where reliable transcripts are unavailable. Traditional evaluation methods often overlook valuable information from the source audio, which can lead to inaccurate assessments. By exploring the use of automatic speech recognition (ASR) transcripts and back-translations, the researchers demonstrate that ASR transcripts provide a more reliable synthetic source. This study is particularly relevant as it encompasses two ST benchmarks covering 79 language pairs and evaluates six ST systems with varying architectures and performance levels. The findings underscore the need for improved evaluation metrics that consider the source audio, paving the way for advancements in machine translation and speech recognition technologies.