Two-Step Knowledge Distillation for Tiny Speech Enhancement
Rayan Daod Nathoo Mikolaj Kegler Marko Stamenovic
Tiny, causal models are crucial for embedded audio machine learning
applications. Model compression can be achieved via distilling knowledge from a
large teacher into a smaller student model. In this work, we propose a novel
two-step approach for tiny speech enhancement model distillation. In contrast
to the standard approach of a weighted mixture of distillation and supervised
losses, we firstly pre-train the student using only the knowledge distillation
(KD) objective, after which we switch to a fully supervised training regime. We
also propose a novel fine-grained similarity-preserving KD loss, which aims to
match the student's intra-activation Gram matrices to that of the teacher. Our
method demonstrates broad improvements, but particularly shines in adverse
conditions including high compression and low signal to noise ratios (SNR),
yielding signal to distortion ratio gains of 0.9 dB and 1.1 dB, respectively,
at -5 dB input SNR and 63x compression compared to baseline.