Assessing Identity Leakage in Talking Face Generation: Metrics and Evaluation Framework

The recent publication on arXiv presents a significant advancement in the field of talking face generation by tackling the challenge of identity leakage, where generated lip movements are improperly influenced by reference images rather than the intended audio. This phenomenon complicates the fidelity of AI-generated videos, making it essential to establish reliable metrics for evaluation. The authors propose a systematic evaluation methodology that includes three complementary test setups: silent-input generation, mismatched audio-video pairing, and matched audio-video synthesis. Additionally, they introduce new metrics such as lip-sync discrepancy and silent-audio-based lip-sync scores to better quantify the leakage. By studying how different identity reference selections impact leakage, the research provides valuable insights into reference design. This model-agnostic methodology sets a new benchmark for future research, ensuring that advancements in AI-generated media can be assess…