Harder Tasks Need More Experts: Dynamic Routing in MoE Models
Quzhe Huang Zhenwei An Nan Zhuang Mingxu Tao Chen Zhang Yang Jin Kun Xu Kun Xu Liwei Chen Songfang Huang Yansong Feng
In this paper, we introduce a novel dynamic expert selection framework for
Mixture of Experts (MoE) models, aiming to enhance computational efficiency and
model performance by adjusting the number of activated experts based on input
difficulty. Unlike traditional MoE approaches that rely on fixed Top-K routing,
which activates a predetermined number of experts regardless of the input's
complexity, our method dynamically selects experts based on the confidence
level in expert selection for each input. This allows for a more efficient
utilization of computational resources, activating more experts for complex
tasks requiring advanced reasoning and fewer for simpler tasks. Through
extensive evaluations, our dynamic routing method demonstrates substantial
improvements over conventional Top-2 routing across various benchmarks,
achieving an average improvement of 0.7% with less than 90% activated
parameters. Further analysis shows our model dispatches more experts to tasks
requiring complex reasoning skills, like BBH, confirming its ability to
dynamically allocate computational resources in alignment with the input's
complexity. Our findings also highlight a variation in the number of experts
needed across different layers of the transformer model, offering insights into
the potential for designing heterogeneous MoE frameworks. The code and models
are available at https://github.com/ZhenweiAn/Dynamic_MoE.