Customizable Combination of Parameter-Efficient Modules for Multi-Task
  Learning
Haowen Wang Tao Sun Cong Fan Jinjie Gu
Modular and composable transfer learning is an emerging direction in the
field of Parameter Efficient Fine-Tuning, as it enables neural networks to
better organize various aspects of knowledge, leading to improved cross-task
generalization. In this paper, we introduce a novel approach Customized
Polytropon C-Poly that combines task-common skills and task-specific skills,
while the skill parameters being highly parameterized using low-rank
techniques. Each task is associated with a customizable number of exclusive
specialized skills and also benefits from skills shared with peer tasks. A
skill assignment matrix is jointly learned. To evaluate our approach, we
conducted extensive experiments on the Super-NaturalInstructions and the
SuperGLUE benchmarks. Our findings demonstrate that C-Poly outperforms
fully-shared, task-specific, and skill-indistinguishable baselines,
significantly enhancing the sample efficiency in multi-task learning scenarios.