Factorised Speaker-environment Adaptive Training of Conformer Speech
  Recognition Systems
Jiajun Deng Guinan Li Xurong Xie Zengrui Jin Mingyu Cui Tianzi Wang Shujie Hu Mengzhe Geng Xunying Liu
Rich sources of variability in natural speech present significant challenges
to current data intensive speech recognition technologies. To model both
speaker and environment level diversity, this paper proposes a novel Bayesian
factorised speaker-environment adaptive training and test time adaptation
approach for Conformer ASR models. Speaker and environment level
characteristics are separately modeled using compact hidden output transforms,
which are then linearly or hierarchically combined to represent any
speaker-environment combination. Bayesian learning is further utilized to model
the adaptation parameter uncertainty. Experiments on the 300-hr WHAM noise
corrupted Switchboard data suggest that factorised adaptation consistently
outperforms the baseline and speaker label only adapted Conformers by up to
3.1% absolute (10.4% relative) word error rate reductions. Further analysis
shows the proposed method offers potential for rapid adaption to unseen
speaker-environment conditions.