LiveTune: Dynamic Parameter Tuning for Training Deep Neural Networks
Soheil Zibakhsh Shabgahi Nojan Sheybani Aiden Tabrizi Farinaz Koushanfar
Traditional machine learning training is a static process that lacks
real-time adaptability of hyperparameters. Popular tuning solutions during
runtime involve checkpoints and schedulers. Adjusting hyper-parameters usually
require the program to be restarted, wasting utilization and time, while
placing unnecessary strain on memory and processors. We present LiveTune, a new
framework allowing real-time parameter tuning during training through
LiveVariables. Live Variables allow for a continuous training session by
storing parameters on designated ports on the system, allowing them to be
dynamically adjusted. Extensive evaluations of our framework show saving up to
60 seconds and 5.4 Kilojoules of energy per hyperparameter change.