Sub-network Discovery and Soft-masking for Continual Learning of Mixed
  Tasks
Zixuan Ke Bing Liu Wenhan Xiong Asli Celikyilmaz Haoran Li
Continual learning (CL) has two main objectives: preventing catastrophic
forgetting (CF) and encouraging knowledge transfer (KT). The existing
literature mainly focused on overcoming CF. Some work has also been done on KT
when the tasks are similar. To our knowledge, only one method has been proposed
to learn a sequence of mixed tasks. However, these techniques still suffer from
CF and/or limited KT. This paper proposes a new CL method to achieve both. It
overcomes CF by isolating the knowledge of each task via discovering a
subnetwork for it. A soft-masking mechanism is also proposed to preserve the
previous knowledge and to enable the new task to leverage the past knowledge to
achieve KT. Experiments using classification, generation, information
extraction, and their mixture (i.e., heterogeneous tasks) show that the
proposed method consistently outperforms strong baselines.