The Unreasonable Effectiveness Of Early Discarding After One Epoch In
  Neural Network Hyperparameter Optimization
Romain Egele Felix Mohr Tom Viering Prasanna Balaprakash
To reach high performance with deep learning, hyperparameter optimization
(HPO) is essential. This process is usually time-consuming due to costly
evaluations of neural networks. Early discarding techniques limit the resources
granted to unpromising candidates by observing the empirical learning curves
and canceling neural network training as soon as the lack of competitiveness of
a candidate becomes evident. Despite two decades of research, little is
understood about the trade-off between the aggressiveness of discarding and the
loss of predictive performance. Our paper studies this trade-off for several
commonly used discarding techniques such as successive halving and learning
curve extrapolation. Our surprising finding is that these commonly used
techniques offer minimal to no added value compared to the simple strategy of
discarding after a constant number of epochs of training. The chosen number of
epochs depends mostly on the available compute budget. We call this approach
i-Epoch (i being the constant number of epochs with which neural networks are
trained) and suggest to assess the quality of early discarding techniques by
comparing how their Pareto-Front (in consumed training epochs and predictive
performance) complement the Pareto-Front of i-Epoch.