Saturn: Efficient Multi-Large-Model Deep Learning
Kabir Nagrecha Arun Kumar
In this paper, we propose Saturn, a new data system to improve the efficiency
of multi-large-model training (e.g., during model selection/hyperparameter
optimization). We first identify three key interconnected systems challenges
for users building large models in this setting -- parallelism technique
selection, distribution of GPUs over jobs, and scheduling. We then formalize
these as a joint problem, and build a new system architecture to tackle these
challenges simultaneously. Our evaluations show that our joint-optimization
approach yields 39-49% lower model selection runtimes than typical current DL
practice.