Dynamic Switch Layers For Unsupervised Learning
Haiguang Li Usama Pervaiz Michał Matuszak Robert Kamara Gilles Roux Trausti Thormundsson Joseph Antognini
On-device machine learning (ODML) enables intelligent applications on
resource-constrained devices. However, power consumption poses a major
challenge, forcing a trade-off between model accuracy and power efficiency that
often limits model complexity. The previously established Gated Compression
(GC) layers offer a solution, enabling power efficiency without sacrificing
model performance by selectively gating samples that lack signals of interest.
However, their reliance on ground truth labels limits GC layers to supervised
tasks. This work introduces the Dynamic Switch Layer (DSL), extending the
benefits of GC layers to unsupervised learning scenarios, and maintaining power
efficiency without the need for labeled data. The DSL builds upon the GC
architecture, leveraging a dynamic pathway selection, and adapting model
complexity in response to the innate structure of the data. We integrate the
DSL into the SoundStream architecture and demonstrate that by routing up to 80%
of samples through a lightweight pass we achieve a 12.3x reduction in the
amount of computation performed and a 20.9x reduction in model size. This
reduces the on-device inference latency by up to 26.5% and improves power
efficiency by up to 21.4% without impacting model performance.