MOSPC: MOS Prediction Based on Pairwise Comparison
Kexin Wang Yunlong Zhao Qianqian Dong Tom Ko Mingxuan Wang
As a subjective metric to evaluate the quality of synthesized speech, Mean
opinion score~(MOS) usually requires multiple annotators to score the same
speech. Such an annotation approach requires a lot of manpower and is also
time-consuming. MOS prediction model for automatic evaluation can significantly
reduce labor cost. In previous works, it is difficult to accurately rank the
quality of speech when the MOS scores are close. However, in practical
applications, it is more important to correctly rank the quality of synthesis
systems or sentences than simply predicting MOS scores. Meanwhile, as each
annotator scores multiple audios during annotation, the score is probably a
relative value based on the first or the first few speech scores given by the
annotator. Motivated by the above two points, we propose a general framework
for MOS prediction based on pair comparison (MOSPC), and we utilize C-Mixup
algorithm to enhance the generalization performance of MOSPC. The experiments
on BVCC and VCC2018 show that our framework outperforms the baselines on most
of the correlation coefficient metrics, especially on the metric KTAU related
to quality ranking. And our framework also surpasses the strong baseline in
ranking accuracy on each fine-grained segment. These results indicate that our
framework contributes to improving the ranking accuracy of speech quality.