Exploring Speech Recognition, Translation, and Understanding with
  Discrete Speech Units: A Comparative Study
Xuankai Chang Brian Yan Kwanghee Choi Jeeweon Jung Yichen Lu Soumi Maiti Roshan Sharma Jiatong Shi Jinchuan Tian Shinji Watanabe Yuya Fujita Takashi Maekaku Pengcheng Guo Yao-Fei Cheng Pavel Denisov Kohei Saijo Hsiu-Hsuan Wang
Speech signals, typically sampled at rates in the tens of thousands per
second, contain redundancies, evoking inefficiencies in sequence modeling.
High-dimensional speech features such as spectrograms are often used as the
input for the subsequent model. However, they can still be redundant. Recent
investigations proposed the use of discrete speech units derived from
self-supervised learning representations, which significantly compresses the
size of speech data. Applying various methods, such as de-duplication and
subword modeling, can further compress the speech sequence length. Hence,
training time is significantly reduced while retaining notable performance. In
this study, we undertake a comprehensive and systematic exploration into the
application of discrete units within end-to-end speech processing models.
Experiments on 12 automatic speech recognition, 3 speech translation, and 1
spoken language understanding corpora demonstrate that discrete units achieve
reasonably good results in almost all the settings. We intend to release our
configurations and trained models to foster future research efforts.