SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech
  Recognition Evaluation
Jiayu Du Jinpeng Li Guoguo Chen Wei-Qiang Zhang
In the wake of the surging tide of deep learning over the past decade,
Automatic Speech Recognition (ASR) has garnered substantial attention, leading
to the emergence of numerous publicly accessible ASR systems that are actively
being integrated into our daily lives. Nonetheless, the impartial and
replicable evaluation of these ASR systems encounters challenges due to various
crucial subtleties. In this paper we introduce the SpeechColab Leaderboard, a
general-purpose, open-source platform designed for ASR evaluation. With this
platform: (i) We report a comprehensive benchmark, unveiling the current
state-of-the-art panorama for ASR systems, covering both open-source models and
industrial commercial services. (ii) We quantize how distinct nuances in the
scoring pipeline influence the final benchmark outcomes. These include nuances
related to capitalization, punctuation, interjection, contraction, synonym
usage, compound words, etc. These issues have gained prominence in the context
of the transition towards an End-to-End future. (iii) We propose a practical
modification to the conventional Token-Error-Rate (TER) evaluation metric, with
inspirations from Kolmogorov complexity and Normalized Information Distance
(NID). This adaptation, called modified-TER (mTER), achieves proper
normalization and symmetrical treatment of reference and hypothesis. By
leveraging this platform as a large-scale testing ground, this study
demonstrates the robustness and backward compatibility of mTER when compared to
TER. The SpeechColab Leaderboard is accessible at
https://github.com/SpeechColab/Leaderboard