HuBERTopic: Enhancing Semantic Representation of HuBERT through
  Self-supervision Utilizing Topic Model
Takashi Maekaku Jiatong Shi Xuankai Chang Yuya Fujita Shinji Watanabe
Recently, the usefulness of self-supervised representation learning (SSRL)
methods has been confirmed in various downstream tasks. Many of these models,
as exemplified by HuBERT and WavLM, use pseudo-labels generated from spectral
features or the model's own representation features. From previous studies, it
is known that the pseudo-labels contain semantic information. However, the
masked prediction task, the learning criterion of HuBERT, focuses on local
contextual information and may not make effective use of global semantic
information such as speaker, theme of speech, and so on. In this paper, we
propose a new approach to enrich the semantic representation of HuBERT. We
apply topic model to pseudo-labels to generate a topic label for each
utterance. An auxiliary topic classification task is added to HuBERT by using
topic labels as teachers. This allows additional global semantic information to
be incorporated in an unsupervised manner. Experimental results demonstrate
that our method achieves comparable or better performance than the baseline in
most tasks, including automatic speech recognition and five out of the eight
SUPERB tasks. Moreover, we find that topic labels include various information
about utterance, such as gender, speaker, and its theme. This highlights the
effectiveness of our approach in capturing multifaceted semantic nuances.