OrchMoE: Efficient Multi-Adapter Learning with Task-Skill Synergy
Haowen Wang Tao Sun Kaixiang Ji Jian Wang Cong Fan Jinjie Gu
We advance the field of Parameter-Efficient Fine-Tuning (PEFT) with our novel
multi-adapter method, OrchMoE, which capitalizes on modular skill architecture
for enhanced forward transfer in neural networks. Unlike prior models that
depend on explicit task identification inputs, OrchMoE automatically discerns
task categories, streamlining the learning process. This is achieved through an
integrated mechanism comprising an Automatic Task Classification module and a
Task-Skill Allocation module, which collectively deduce task-specific
classifications and tailor skill allocation matrices. Our extensive evaluations
on the 'Super Natural Instructions' dataset, featuring 1,600 diverse
instructional tasks, indicate that OrchMoE substantially outperforms comparable
multi-adapter baselines in terms of both performance and sample utilization
efficiency, all while operating within the same parameter constraints. These
findings suggest that OrchMoE offers a significant leap forward in multi-task
learning efficiency.