Matching Latent Encoding for Audio-Text based Keyword Spotting
Kumari Nishu Minsik Cho Devang Naik
Using audio and text embeddings jointly for Keyword Spotting (KWS) has shown
high-quality results, but the key challenge of how to semantically align two
embeddings for multi-word keywords of different sequence lengths remains
largely unsolved. In this paper, we propose an audio-text-based end-to-end
model architecture for flexible keyword spotting (KWS), which builds upon
learned audio and text embeddings. Our architecture uses a novel dynamic
programming-based algorithm, Dynamic Sequence Partitioning (DSP), to optimally
partition the audio sequence into the same length as the word-based text
sequence using the monotonic alignment of spoken content. Our proposed model
consists of an encoder block to get audio and text embeddings, a projector
block to project individual embeddings to a common latent space, and an
audio-text aligner containing a novel DSP algorithm, which aligns the audio and
text embeddings to determine if the spoken content is the same as the text.
Experimental results show that our DSP is more effective than other
partitioning schemes, and the proposed architecture outperformed the
state-of-the-art results on the public dataset in terms of Area Under the ROC
Curve (AUC) and Equal-Error-Rate (EER) by 14.4 % and 28.9%, respectively.