Shortcut-connected Expert Parallelism for Accelerating
  Mixture-of-Experts
Weilin Cai Juyong Jiang Le Qin Junwei Cui Sunghun Kim Jiayi Huang
Expert parallelism has been introduced as a strategy to distribute the
computational workload of sparsely-gated mixture-of-experts (MoE) models across
multiple computing devices, facilitating the execution of these increasingly
large-scale models. However, the All-to-All communication intrinsic to expert
parallelism constitutes a significant overhead, diminishing the MoE models'
efficiency. Current optimization approaches offer some relief, yet they are
constrained by the sequential interdependence of communication and computation
operations. To address this limitation, we present a novel shortcut-connected
MoE architecture with overlapping parallel strategy, designated as ScMoE, which
effectively decouples communication from its conventional sequence, allowing
for a substantial overlap of 70% to 100% with computation. When compared with
the prevalent top-2 MoE architecture, ScMoE demonstrates training speed
improvements of 30% and 11%, and inference improvements of 40% and 15%, in our
PCIe and NVLink hardware environments, respectively, where communication
constitutes 60% and 15% of the total MoE time consumption. On the other hand,
extensive experiments and theoretical analyses indicate that ScMoE not only
achieves comparable but in some instances surpasses the model quality of
existing approaches in vision and language tasks.