Diversifying the Mixture-of-Experts Representation for Language Models
  with Orthogonal Optimizer
Boan Liu Liang Ding Li Shen Keqin Peng Yu Cao Dazhao Cheng Dacheng Tao
The Mixture of Experts (MoE) has emerged as a highly successful technique in
deep learning, based on the principle of divide-and-conquer to maximize model
capacity without significant additional computational cost. Even in the era of
large-scale language models (LLMs), MoE continues to play a crucial role, as
some researchers have indicated that GPT-4 adopts the MoE structure to ensure
diverse inference results. However, MoE is susceptible to performance
degeneracy, particularly evident in the issues of imbalance and homogeneous
representation among experts. While previous studies have extensively addressed
the problem of imbalance, the challenge of homogeneous representation remains
unresolved. In this study, we shed light on the homogeneous representation
problem, wherein experts in the MoE fail to specialize and lack diversity,
leading to frustratingly high similarities in their representations (up to 99%
in a well-performed MoE model). This problem restricts the expressive power of
the MoE and, we argue, contradicts its original intention. To tackle this
issue, we propose a straightforward yet highly effective solution: OMoE, an
orthogonal expert optimizer. Additionally, we introduce an alternating training
strategy that encourages each expert to update in a direction orthogonal to the
subspace spanned by other experts. Our algorithm facilitates MoE training in
two key ways: firstly, it explicitly enhances representation diversity, and
secondly, it implicitly fosters interaction between experts during orthogonal
weights computation. Through extensive experiments, we demonstrate that our
proposed optimization algorithm significantly improves the performance of
fine-tuning the MoE model on the GLUE benchmark, SuperGLUE benchmark,
question-answering task, and name entity recognition tasks.