ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference
Hyungjun Oh Kihong Kim Jaemin Kim Sungkyun Kim Junyeol Lee Du-seong Chang Jiwon Seo
This paper presents ExeGPT, a distributed system designed for
constraint-aware LLM inference. ExeGPT finds and runs with an optimal execution
schedule to maximize inference throughput while satisfying a given latency
constraint. By leveraging the distribution of input and output sequences, it
effectively allocates resources and determines optimal execution
configurations, including batch sizes and partial tensor parallelism. We also
introduce two scheduling strategies based on Round-Robin Allocation and
Workload-Aware Allocation policies, suitable for different NLP workloads. We
evaluate ExeGPT on six LLM instances of T5, OPT, and GPT-3 and five NLP tasks,
each with four distinct latency constraints. Compared to FasterTransformer,
ExeGPT achieves up to 15.2x improvements in throughput and 6x improvements in
latency. Overall, ExeGPT achieves an average throughput gain of 2.9x across
twenty evaluation scenarios. Moreover, when adapting to changing sequence
distributions, the cost of adjusting the schedule in ExeGPT is reasonably
modest. ExeGPT proves to be an effective solution for optimizing and executing
LLM inference for diverse NLP workload and serving conditions.