LibriSpeech-PC: Benchmark for Evaluation of Punctuation and
  Capitalization Capabilities of end-to-end ASR Models
Aleksandr Meister Matvei Novikov Nikolay Karpov Evelina Bakhturina Vitaly Lavrukhin Boris Ginsburg
Traditional automatic speech recognition (ASR) models output lower-cased
words without punctuation marks, which reduces readability and necessitates a
subsequent text processing model to convert ASR transcripts into a proper
format. Simultaneously, the development of end-to-end ASR models capable of
predicting punctuation and capitalization presents several challenges,
primarily due to limited data availability and shortcomings in the existing
evaluation methods, such as inadequate assessment of punctuation prediction. In
this paper, we introduce a LibriSpeech-PC benchmark designed to assess the
punctuation and capitalization prediction capabilities of end-to-end ASR
models. The benchmark includes a LibriSpeech-PC dataset with restored
punctuation and capitalization, a novel evaluation metric called Punctuation
Error Rate (PER) that focuses on punctuation marks, and initial baseline
models. All code, data, and models are publicly available.