Batching for Green AI -- An Exploratory Study on Inference
Tim Yarally Luís Cruz Daniel Feitosa June Sallou Arie van Deursen
The batch size is an essential parameter to tune during the development of
new neural networks. Amongst other quality indicators, it has a large degree of
influence on the model's accuracy, generalisability, training times and
parallelisability. This fact is generally known and commonly studied. However,
during the application phase of a deep learning model, when the model is
utilised by an end-user for inference, we find that there is a disregard for
the potential benefits of introducing a batch size. In this study, we examine
the effect of input batching on the energy consumption and response times of
five fully-trained neural networks for computer vision that were considered
state-of-the-art at the time of their publication. The results suggest that
batching has a significant effect on both of these metrics. Furthermore, we
present a timeline of the energy efficiency and accuracy of neural networks
over the past decade. We find that in general, energy consumption rises at a
much steeper pace than accuracy and question the necessity of this evolution.
Additionally, we highlight one particular network, ShuffleNetV2(2018), that
achieved a competitive performance for its time while maintaining a much lower
energy consumption. Nevertheless, we highlight that the results are model
dependent.