Enhancing Gappy Speech Audio Signals with Generative Adversarial
  Networks
Deniss Strods Alan F. Smeaton
Gaps, dropouts and short clips of corrupted audio are a common problem and
particularly annoying when they occur in speech. This paper uses machine
learning to regenerate gaps of up to 320ms in an audio speech signal. Audio
regeneration is translated into image regeneration by transforming audio into a
Mel-spectrogram and using image in-painting to regenerate the gaps. The full
Mel-spectrogram is then transferred back to audio using the Parallel-WaveGAN
vocoder and integrated into the audio stream. Using a sample of 1300 spoken
audio clips of between 1 and 10 seconds taken from the publicly-available
LJSpeech dataset our results show regeneration of audio gaps in close to real
time using GANs with a GPU equipped system. As expected, the smaller the gap in
the audio, the better the quality of the filled gaps. On a gap of 240ms the
average mean opinion score (MOS) for the best performing models was 3.737, on a
scale of 1 (worst) to 5 (best) which is sufficient for a human to perceive as
close to uninterrupted human speech.