A Codesign of Scheduling and Parallelization for Large Model Training in
  Heterogeneous Clusters
Chunyu Xue Weihao Cui Han Zhao Quan Chen Shulai Zhang Pengyu Yang Jing Yang Shaobo Li Minyi Guo
Joint consideration of scheduling and adaptive parallelism offers great
opportunities for improving the training efficiency of large models on
heterogeneous GPU clusters. However, integrating adaptive parallelism into a
cluster scheduler expands the cluster scheduling space. The new space is the
product of the original scheduling space and the parallelism exploration space
of adaptive parallelism (also a product of pipeline, data, and tensor
parallelism). The exponentially enlarged scheduling space and ever-changing
optimal parallelism plan from adaptive parallelism together result in the
contradiction between low-overhead and accurate performance data acquisition
for efficient cluster scheduling. This paper presents Crius, a training system
for efficiently scheduling multiple large models with adaptive parallelism in a
heterogeneous cluster. Crius proposes a novel scheduling granularity called
Cell. It represents a job with deterministic resources and pipeline stages. The
exploration space of Cell is shrunk to the product of only data and tensor
parallelism, thus exposing the potential for accurate and low-overhead
performance estimation. Crius then accurately estimates Cells and efficiently
schedules training jobs. When a Cell is selected as a scheduling choice, its
represented job runs with the optimal parallelism plan explored. Experimental
results show that Crius reduces job completion time by up to 48.9% and
schedules large models with up to 1.49x cluster throughput improvement.