This paper introduces HybridNet, a hybrid neural network to speed-up autoregressive
 models for raw audio waveform generation. As an example, we propose
 a hybrid model that combines an autoregressive network named WaveNet and a
 conventional LSTM model to address speech synthesis. Instead of generating
 one sample per time-step, the proposed HybridNet generates multiple samples per
 time-step by exploiting the long-term memory utilization property of LSTMs. In
 the evaluation, when applied to text-to-speech, HybridNet yields state-of-art performance.
 HybridNet achieves a 3.83 subjective 5-scale mean opinion score on
 US English, largely outperforming the same size WaveNet in terms of naturalness
 and provide 2x speed up at inference. Speech synthesis, also known as text-to-speech (TTS) has variety of applications in human-computer interactions, assistive technology and entertainment productions. The traditional TTS system, which is done with complex hand-engineered pipelines, transforms textual features into high temporal resolution waveforms (e.g., 16KHz). Recent work on neural TTS has demonstrated state-of-the-art results BID0 b; BID14 . In particular, various neural network architectures (e.g., BID12 have been proposed as neural vocoders for waveform synthesis.Recurrent neural network (RNN), especially long short-term memory (LSTM) BID6 , is well-suited to address speech synthesis as it can model long-term dependencies in audio data (e.g., BID3 BID16 BID4 . RNNs have been successfully applied in many state-of-the-art neural TTS systems (e.g., BID14 BID1 , and were proven to be more effective than the conventional hidden Markov model (HMM)-based synthesizer. However, RNNs are unsuitable for raw waveform generation at high sampling rate (e.g. 16,000 samples per second), because RNNs process each state sequentially and the computation cannot be paralleled over elements of a sequence at training. In practice, RNNs are usually operated on spectral or hand-engineered features of audio (e.g., BID14 BID5 , which have much fewer time steps than the raw waveform. Most recently, SampleRNN was proposed to tackle this difficulty by combining autoregressive multilayer perceptrons and RNNs in a hierarchical architecture, where different RNN modules operate at different clock rates.Another line of research investigates the convolutional autoregressive model (e.g., WaveNet BID12 ) for waveform synthesis, where the computation over different time-step can be fully parallelized during training. WaveNet can be efficiently trained on audio data with tens of thousands of samples per second. In order to model the long-range dependencies in audio data, WaveNet uses dilated convolution BID15 to increase the receptive fields of output units, and demonstrates very good performance in speech synthesis. Several state-of-the-art neural TTS systems, such as Deep Voice 1 BID0 and Deep Voice 2 BID1 , use WaveNet to synthesize waveform conditioned on acoustic features (e.g., phoneme duration and fundamental frequency). Despite its full parallelism at training, WaveNet poses daunting computational problem at inference due to the autoregressive nature of the model (see FIG0 for an illustration).In addition, although the dilated convolution is very effective at increasing receptive fields, the very long-range connections in deep layer could potentially result in high variance in the output sampling distribution. In practice, some buzz noises are commonly observed in the audio samples generated by WaveNet. We present a hybrid neural architecture to speed up autoregressive models for audio synthesis. As a demonstration of effectiveness, we design and implement a hybrid model that consists of an autoregressive network named WaveNet and a LSTM model. The hybrid model exploits the long-term memory utilization property and short inference time of LSTMs. We evaluate HybridNet with both Mean Opinion Score (MOS) and validation error, and find that it outperforms WaveNet with the same layer in terms of naturalness and validation error. In addition, HybridNet provides 2x-4x speed-up at inference with comparable generation quality compared to WaveNet. The technique used in HybridNet can be easily applied to another autoregressive model and can be combined with existing inference time reduction techniques.
It is a hybrid neural architecture to speed-up autoregressive model.  Concludes that in order to scale up the model size without increasing inference time for sequential prediction, use a model that predicts multiple timesteps at once. This paper presents HybridNet, a neural speech and other audio synthesis system that combines the WaveNet model with an LSTM with the goal of offering a model with faster inference-time audio generation.