Distilling Universal and Joint Knowledge for Cross-Domain Model
  Compression on Time Series Data
Qing Xu Min Wu Xiaoli Li Kezhi Mao Zhenghua Chen
For many real-world time series tasks, the computational complexity of
prevalent deep leaning models often hinders the deployment on resource-limited
environments (e.g., smartphones). Moreover, due to the inevitable domain shift
between model training (source) and deploying (target) stages, compressing
those deep models under cross-domain scenarios becomes more challenging.
Although some of existing works have already explored cross-domain knowledge
distillation for model compression, they are either biased to source data or
heavily tangled between source and target data. To this end, we design a novel
end-to-end framework called Universal and joint knowledge distillation (UNI-KD)
for cross-domain model compression. In particular, we propose to transfer both
the universal feature-level knowledge across source and target domains and the
joint logit-level knowledge shared by both domains from the teacher to the
student model via an adversarial learning scheme. More specifically, a
feature-domain discriminator is employed to align teacher's and student's
representations for universal knowledge transfer. A data-domain discriminator
is utilized to prioritize the domain-shared samples for joint knowledge
transfer. Extensive experimental results on four time series datasets
demonstrate the superiority of our proposed method over state-of-the-art (SOTA)
benchmarks.