  Residual and skip connections play an important role in many current
  generative models. Although their theoretical and numerical advantages
  are understood, their role in speech enhancement systems has not been
  investigated so far. When performing spectral speech enhancement,
  residual connections are very similar in nature to spectral subtraction,
  which is the one of the most commonly employed speech enhancement approaches.
   Highway networks, on the other hand, can be seen as a combination of spectral
  masking and spectral subtraction. However, when using deep neural networks, such operations would
  normally happen in a transformed spectral domain, as opposed to traditional speech
  enhancement where all operations are often done directly on the spectrum.
   In this paper, we aim to investigate the role of residual and highway
  connections in deep neural networks for speech enhancement, and verify whether
  or not they operate similarly to their traditional, digital signal processing
  counterparts. We visualize the outputs of such connections, projected back to
  the spectral domain, in models trained for speech denoising, and show that while
  skip connections do not necessarily improve performance with regards to the
  number of parameters, they make speech enhancement models more interpretable. Highway BID7 and residual networks BID1 have been proposed with the objective of improving activation and gradient flow in the training of deep neural networks. On the other hand, in tasks like image reconstruction or speech enhancement, the use of such skip connections serves a different purpose: if we model a corrupted signal x = y + n as the addition of noise n to a clean signal y and x is the input to a neural network, we know that the task at hand is to predict n. In other words, to predict y, we have to alter the input x by subtracting n.In speech enhancement, the two more commonly used approaches are spectral subtraction and spectral masking. In the first, a statistical model of n is used to predict its magnitude spectrum N , which is then subtracted from the input spectrum X to yield a clean magnitude spectrum estimateŶ . In spectral masking, instead of performing subtraction, we find a multiplicative mask M which aims at either blocking time-frequency cells dominated by noise (in the case of binary masks) or scaling down energies in such time-frequency cells to make them match that of the original clean signal. Recent work in speech enhancement has explored skip connections as a way of performing masking BID4 and spectral estimation BID6 . Time domain approaches, such as SEGAN BID5 , use a UNet-style network which employs multiple skip connections as well. Other works, such as BID12 , perform spectral masking but Figure 1: Diagrams for highway, residual, and masking blocks used in this paper learn how to estimate an ideal mask instead of having the masking mechanism embedded in the neural network as a skip connection.For better understanding of such models, we would like to understand whether there are any parallels between such connections and two traditional DSP approaches to speech enhancement, namely spectral subtraction and spectral masking. We also want to understand whether models using skip connections perform better for enhancement when such connections appear only once (resembling their DSP counterparts) or repeated as multiple blocks (like in highway and residual networks). This paper shows early results of our investigation on the role of skip connections in speech enhancement models. Our preliminary experiments show that, although they have no significant impact in the performance of the models, such connections might help making the models more interpretable, as we can identify the contribution of each individual layer to the task. In the future, we intend to investigate more complex models, such as models based on the UNet architecture, as well as models that employ a temporal context window at the input instead of a single frame (such as the work in BID6 ), since those are more in line with state-of-the-art models in the literature.
We show how using skip connections can make speech enhancement models more interpretable, as it makes them use similar mechanisms that have been explored in the DSP literature. The authors propose incorporating Residual, Highway and Masking blocks inside a fully convolutional pipeline in order to understand how iterative inference of the output and the masking is performed in a speech enhancement task The authors interpret highway, residual and masking connections.  The authors generate their own noisy speech by artificially adding noise from a well established noise data-set to a less know clean speech data-set.