In this paper we introduce a new speech recognition system, leveraging a simple letter-based ConvNet acoustic model. The acoustic model requires only audio transcription for training -- no alignment annotations, nor any forced alignment step is needed. At inference, our decoder takes only a word list and a language model, and is fed with letter scores from the acoustic model -- no phonetic word lexicon is needed. Key ingredients for the acoustic model are Gated Linear Units and high dropout. We show near state-of-the-art results in word error rate on the LibriSpeech corpus with MFSC features, both on the clean and other configurations.
 Top speech recognition systems are either complicated pipelines or using more data that is publicly available. We set out to show that it is possible to train a nearly state of the art speech recognition system for read speech, with a public dataset (LibriSpeech), on a GPU-equipped workstation. Thus, we present an end-to-end system for speech recognition, going from Mel-Frequency Spectral Coefficients (MFSCs) to the transcription in words. The acoustic model is trained using letters (graphemes) directly, which take out the need for an intermediate (human or automatic) phonetic transcription.The classical pipeline to build state of the art systems for speech recognition consists in first training an HMM/GMM model to force align the units on which the final acoustic model operates (most often context-dependent phone states). This approach takes its roots in HMM/GMM training BID31 . The improvements brought by deep neural networks (DNNs) and convolutional neural networks (CNNs) BID26 BID27 for acoustic modeling only extend this training pipeline. The current state of the art on LibriSpeech belongs to this approach too BID16 BID19 , with an additional step of speaker adaptation BID22 BID18 . Recently, BID25 proposed GMM-free training, but the approach still requires to generate a forced alignment.An approach that cut ties with the HMM/GMM pipeline (and with forced alignment) was to train with a recurrent neural network (RNN) BID7 ) for phoneme transcription. There are now competitive end-to-end approaches of acoustic models toppled with RNNs layers as in BID9 BID14 BID23 BID0 , trained with a sequence criterion BID8 . However these models are computationally expensive, and thus often take a long time to train. On conversational speech (that is not the topic of this paper), the state of the art is still held by complex ConvNets+RNNs acoustic models, coupled to domain-adapted language models BID32 BID24 .Compared to classical approaches that need phonetic annotation (often derived from a phonetic dictionary, rules, and generative training), we propose to train the model end-to-end, using graphemes directly. Compared to sequence criterion based approaches that train directly from speech signal to graphemes BID14 , we propose an RNN-free architecture based on convolutional networks for the acoustic model, toppled with a simple sequence-level variant of CTC.We reach the clean speech performance of BID19 , but without performing speaker adaptation. Our word-error-rate on clean speech is better than BID0 , while being worse on noisy speech, but they train on 11,900 hours while we only train on the 960h available in LibriSpeech's train set. The rest of the paper is structured as follows: the next section presents the convolutional networks used for acoustic modeling, along with the automatic segmentation criterion and decoding approaches. The last section shows experimental results on LibriSpeech. Figure 1: Overview of our acoustic model, which computes MFSC features which are fed to a Gated ConvNet. The ConvNet output one score for each letter in the dictionary, and for each MFSC frame. At inference time, theses scores are fed to a decoder (see Section 2.4) to form the most likely sequence of words. At training time, the scores are fed to the ASG criterion (see FIG1 ) which promotes sequences of letters leading to the transcrition sequence (here "c a t"). We have introduced a simple end-to-end automatic speech recognition system, which combines a large (208M parameters) but efficient ConvNet acoustic model, an easy sequence criterion which can infer the segmentation, and a simple beam-search decoder. The decoding results are competitive on the LibriSpeech corpus (4.8% WER dev-clean). Our approach breaks free from HMM/GMM pre-training and forced alignment, as well as not being as computationally intensive as RNN-based approaches BID0 . We based all our work on a publicly available (free) dataset, all of which should make it easier to reproduce. Further work should include leveraging speaker identity, training from the raw waveform, data augmentation, training with more data, better language models.
A letter-based ConvNet acoustic model leads to a simple and competitive speech recognition pipeline. This paper applies gated convolutional neural networks to speech recognition, using the training criterion ASG.