Dual Knowledge Distillation for Efficient Sound Event Detection
Yang Xiao Rohan Kumar Das
Sound event detection (SED) is essential for recognizing specific sounds and
their temporal locations within acoustic signals. This becomes challenging
particularly for on-device applications, where computational resources are
limited. To address this issue, we introduce a novel framework referred to as
dual knowledge distillation for developing efficient SED systems in this work.
Our proposed dual knowledge distillation commences with temporal-averaging
knowledge distillation (TAKD), utilizing a mean student model derived from the
temporal averaging of the student model's parameters. This allows the student
model to indirectly learn from a pre-trained teacher model, ensuring a stable
knowledge distillation. Subsequently, we introduce embedding-enhanced feature
distillation (EEFD), which involves incorporating an embedding distillation
layer within the student model to bolster contextual learning. On DCASE 2023
Task 4A public evaluation dataset, our proposed SED system with dual knowledge
distillation having merely one-third of the baseline model's parameters,
demonstrates superior performance in terms of PSDS1 and PSDS2. This highlights
the importance of proposed dual knowledge distillation for compact SED systems,
which can be ideal for edge devices.