Perseus: Removing Energy Bloat from Large Model Training
Jae-Won Chung Yile Gu Insu Jang Luoxi Meng Nikhil Bansal Mosharaf Chowdhury
Training large AI models on numerous GPUs consumes a massive amount of
energy. We observe that not all energy consumed during training directly
contributes to end-to-end training throughput, and a significant portion can be
removed without slowing down training, which we call energy bloat.
  In this work, we identify two independent sources of energy bloat in large
model training, intrinsic and extrinsic, and propose Perseus, a unified
optimization framework that mitigates both. Perseus obtains the "iteration
time-energy" Pareto frontier of any large model training job using an efficient
iterative graph cut-based algorithm and schedules energy consumption of its
forward and backward computations across time to remove intrinsic and extrinsic
energy bloat. Evaluation on large models like GPT-3 and Bloom shows that
Perseus reduces energy consumption of large model training by up to 30%,
enabling savings otherwise unobtainable before.