We present Optimal Completion Distillation (OCD), a training procedure for optimizing sequence to sequence models based on edit distance. OCD is efficient, has no hyper-parameters of its own, and does not require pre-training or joint optimization with conditional log-likelihood. Given a partial sequence generated by the model, we first identify the set of optimal suffixes that minimize the total edit distance, using an efficient dynamic programming algorithm.   Then, for each position of the generated sequence, we use a target distribution which puts equal probability on the first token of all the optimal suffixes. OCD achieves the state-of-the-art performance on end-to-end speech recognition, on both Wall Street Journal and Librispeech datasets, achieving $9.3\%$ WER and $4.5\%$ WER, respectively. Recent advances in natural language processing and speech recognition hinge on the development of expressive neural network architectures for sequence to sequence (seq2seq) learning BID54 BID1 . Such encoder-decoder architectures are adopted in both machine translation BID1 BID24 and speech recognition systems BID7 BID2 BID11 achieving impressive performance above traditional multi-stage pipelines BID29 BID41 . Improving the building blocks of seq2seq models can fundamentally advance machine translation and speech recognition, and positively impact other domains such as image captioning BID62 , parsing , summarization BID47 , and program synthesis BID65 .To improve the key components of seq2seq models, one can either design better architectures, or develop better learning algorithms. Recent architectures using convolution BID20 and self attention BID57 have proved to be useful, especially to facilitate efficient training. On the other hand, despite many attempts to mitigate the limitations of Maximum Likelihood Estimation (MLE) BID43 BID60 BID4 BID30 , MLE is still considered the dominant approach for training seq2seq models. Current alternative approaches require pre-training or joint optimization with conditional log-likelihood. They are difficult to implement and require careful tuning of new hyper-parameters (e.g. mixing ratios). In addition , alternative approaches typically do not offer a substantial performance improvement over a well tuned MLE baseline, especially when label smoothing BID40 BID18 and scheduled sampling are used.In this paper, we borrow ideas from search-based structured prediction BID15 BID46 and policy distillation BID48 and develop an efficient algorithm for optimizing seq2seq models based on edit distance 1 . Our key observation is that given an arbitrary prefix (e.g. a partial sequence generated by sampling from the model), we can exactly and efficiently identify all of the suffixes that result in a minimum total edit distance (v.s. the ground truth target). Our training procedure , called Optimal Completion Distillation (OCD), is summarized as follows:The proposed OCD algorithm is efficient, straightforward to implement, and has no tunable hyperparameters of its own. Our key contributions include:• We propose OCD, a stand-alone algorithm for optimizing seq2seq models based on edit distance. OCD is scalable to real-world datasets with long sequences and large vocabularies, and consistently outperforms Maximum Likelihood Estimation (MLE) by a large margin.• Given a target sequence of length m and a generated sequence of length n, we present an O(nm) algorithm that identifies all of the optimal extensions for each prefix of the generated sequence.• We demonstrate the effectiveness of OCD on end-to-end speech recognition using attentionbased seq2seq models. On the Wall Street Journal dataset, OCD achieves a Character Error Rate (CER) of 3.1% and a Word Error Rate (WER) of 9.3% without language model rescoring, outperforming all prior work TAB1 . On Librispeech, OCD achieves state-of-the-art WER of 4.5% on "test-clean" and 13.3% on "test-other" sets TAB2 ).
Optimal Completion Distillation (OCD) is a training procedure for optimizing sequence to sequence models based on edit distance which achieves state-of-the-art on end-to-end Speech Recognition tasks. Alternative approach to training seq2seq models using a dynamic program to compute optimal continuations of predicted prefixes A training algorithm for auto-regressive models that does not require any MLE pre-training and can directly optimize from the sampling. The paper considers a shortcoming of sequence to sequence models trained using maximum likelihood estimation and propose an approach based on edit distances and the implicit use of given label sequences during training