Impact of HPO on AutoML Forecasting Ensembles
David Hoffmann
A forecasting ensemble consisting of a diverse range of estimators for both
local and global univariate forecasting, in particular MQ-CNN,DeepAR, Prophet,
NPTS, ARIMA and ETS, can be used to make forecasts for a variety of problems.
This paper delves into the aspect of adding different hyperparameter
optimization strategies to the deep learning models in such a setup (DeepAR and
MQ-CNN), exploring the trade-off between added training cost and the increase
in accuracy for different configurations. It shows that in such a setup, adding
hyperparameter optimization can lead to performance improvements, with the
final setup having a 9.9 % percent accuracy improvement with respect to the
avg-wQL over the baseline ensemble without HPO, accompanied by a 65.8 %
increase in end-to-end ensemble latency. This improvement is based on an
empirical analysis of combining the ensemble pipeline with different tuning
strategies, namely Bayesian Optimisation and Hyperband and different
configurations of those strategies. In the final configuration, the proposed
combination of ensemble learning and HPO outperforms the state of the art
commercial AutoML forecasting solution, Amazon Forecast, with a 3.5 % lower
error and 16.0 % lower end-to-end ensemble latency.