Self-Supervised Speech Quality Estimation and Enhancement Using Only
  Clean Speech
Szu-Wei Fu Kuo-Hsuan Hung Yu Tsao Yu-Chiang Frank Wang
Speech quality estimation has recently undergone a paradigm shift from
human-hearing expert designs to machine-learning models. However, current
models rely mainly on supervised learning, which is time-consuming and
expensive for label collection. To solve this problem, we propose VQScore, a
self-supervised metric for evaluating speech based on the quantization error of
a vector-quantized-variational autoencoder (VQ-VAE). The training of VQ-VAE
relies on clean speech; hence, large quantization errors can be expected when
the speech is distorted. To further improve correlation with real quality
scores, domain knowledge of speech processing is incorporated into the model
design. We found that the vector quantization mechanism could also be used for
self-supervised speech enhancement (SE) model training. To improve the
robustness of the encoder for SE, a novel self-distillation mechanism combined
with adversarial training is introduced. In summary, the proposed speech
quality estimation method and enhancement models require only clean speech for
training without any label requirements. Experimental results show that the
proposed VQScore and enhancement model are competitive with supervised
baselines. The code will be released after publication.