SiDA: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable
  Large Mixture-of-Experts Models
Zhixu Du Shiyu Li Yuhao Wu Xiangyu Jiang Jingwei Sun Qilin Zheng Yongkai Wu Ang Li Hai "Helen" Li Yiran Chen
Mixture-of-Experts (MoE) has emerged as a favorable architecture in the era
of large models due to its inherent advantage, i.e., enlarging model capacity
without incurring notable computational overhead. Yet, the realization of such
benefits often results in ineffective GPU memory utilization, as large portions
of the model parameters remain dormant during inference. Moreover, the memory
demands of large models consistently outpace the memory capacity of
contemporary GPUs. Addressing this, we introduce SiDA (Sparsity-inspired
Data-Aware), an efficient inference approach tailored for large MoE models.
SiDA judiciously exploits both the system's main memory, which is now abundant
and readily scalable, and GPU memory by capitalizing on the inherent sparsity
on expert activation in MoE models. By adopting a data-aware perspective, SiDA
achieves enhanced model efficiency with a neglectable performance drop.
Specifically, SiDA attains a remarkable speedup in MoE inference with up to
3.93X throughput increasing, up to 75% latency reduction, and up to 80% GPU
memory saving with down to 1% performance drop. This work paves the way for
scalable and efficient deployment of large MoE models, even in
memory-constrained systems.