Evaluating Speech Synthesis by Training Recognizers on Synthetic Speech
Dareen Alharthi Roshan Sharma Hira Dhamyal Soumi Maiti Bhiksha Raj Rita Singh
Modern speech synthesis systems have improved significantly, with synthetic
speech being indistinguishable from real speech. However, efficient and
holistic evaluation of synthetic speech still remains a significant challenge.
Human evaluation using Mean Opinion Score (MOS) is ideal, but inefficient due
to high costs. Therefore, researchers have developed auxiliary automatic
metrics like Word Error Rate (WER) to measure intelligibility. Prior works
focus on evaluating synthetic speech based on pre-trained speech recognition
models, however, this can be limiting since this approach primarily measures
speech intelligibility. In this paper, we propose an evaluation technique
involving the training of an ASR model on synthetic speech and assessing its
performance on real speech. Our main assumption is that by training the ASR
model on the synthetic speech, the WER on real speech reflects the similarity
between distributions, a broader assessment of synthetic speech quality beyond
intelligibility. Our proposed metric demonstrates a strong correlation with
both MOS naturalness and MOS intelligibility when compared to SpeechLMScore and
MOSNet on three recent Text-to-Speech (TTS) systems: MQTTS, StyleTTS, and
YourTTS.