On the Efficiency of Convolutional Neural Networks
Andrew Lavin
Since the breakthrough performance of AlexNet in 2012, convolutional neural
networks (convnets) have grown into extremely powerful vision models. Deep
learning researchers have used convnets to produce accurate results that were
unachievable a decade ago. Yet computer scientists make computational
efficiency their primary objective. Accuracy with exorbitant cost is not
acceptable; an algorithm must also minimize its computational requirements.
Confronted with the daunting computation that convnets use, deep learning
researchers also became interested in efficiency. Researchers applied
tremendous effort to find the convnet architectures that have the greatest
efficiency. However, skepticism grew among researchers and engineers alike
about the relevance of arithmetic complexity. Contrary to the prevailing view
that latency and arithmetic complexity are irreconcilable, a simple formula
relates both through computational efficiency. This insight enabled us to
co-optimize the separate factors that determine latency. We observed that the
degenerate conv2d layers that produce the best accuracy-complexity trade-off
also have low operational intensity. Therefore, kernels that implement these
layers use significant memory resources. We solved this optimization problem
with block-fusion kernels that implement all layers of a residual block,
thereby creating temporal locality, avoiding communication, and reducing
workspace size. Our ConvFirst model with block-fusion kernels ran approximately
four times as fast as the ConvNeXt baseline with PyTorch Inductor, at equal
accuracy on the ImageNet-1K classification task. Our unified approach to
convnet efficiency envisions a new era of models and kernels that achieve
greater accuracy at lower cost.