Generalization Error Analysis for Sparse Mixture-of-Experts: A
  Preliminary Study
Jinze Zhao Peihao Wang Zhangyang Wang
Mixture-of-Experts (MoE) represents an ensemble methodology that amalgamates
predictions from several specialized sub-models (referred to as experts). This
fusion is accomplished through a router mechanism, dynamically assigning
weights to each expert's contribution based on the input data. Conventional MoE
mechanisms select all available experts, incurring substantial computational
costs. In contrast, Sparse Mixture-of-Experts (Sparse MoE) selectively engages
only a limited number, or even just one expert, significantly reducing
computation overhead while empirically preserving, and sometimes even
enhancing, performance. Despite its wide-ranging applications and these
advantageous characteristics, MoE's theoretical underpinnings have remained
elusive. In this paper, we embark on an exploration of Sparse MoE's
generalization error concerning various critical factors. Specifically, we
investigate the impact of the number of data samples, the total number of
experts, the sparsity in expert selection, the complexity of the routing
mechanism, and the complexity of individual experts. Our analysis sheds light
on \textit{how \textbf{sparsity} contributes to the MoE's generalization},
offering insights from the perspective of classical learning theory.