End-to-end Joint Rich and Normalized ASR with a limited amount of rich
  training data
Can Cui Imran Ahamad Sheikh Mostafa Sadeghi Emmanuel Vincent
Joint rich and normalized automatic speech recognition (ASR), that produces
transcriptions both with and without punctuation and capitalization, remains a
challenge. End-to-end (E2E) ASR models offer both convenience and the ability
to perform such joint transcription of speech. Training such models requires
paired speech and rich text data, which is not widely available. In this paper,
we compare two different approaches to train a stateless Transducer-based E2E
joint rich and normalized ASR system, ready for streaming applications, with a
limited amount of rich labeled data. The first approach uses a language model
to generate pseudo-rich transcriptions of normalized training data. The second
approach uses a single decoder conditioned on the type of the output. The first
approach leads to E2E rich ASR which perform better on out-of-domain data, with
up to 9% relative reduction in errors. The second approach demonstrates the
feasibility of an E2E joint rich and normalized ASR system using as low as 5%
rich training data with moderate (2.42% absolute) increase in errors.