An automated approach for improving the inference latency and energy
  efficiency of pretrained CNNs by removing irrelevant pixels with focused
  convolutions
Caleb Tung Nicholas Eliopoulos Purvish Jajal Gowri Ramshankar Chen-Yun Yang Nicholas Synovic Xuecen Zhang Vipin Chaudhary George K. Thiruvathukal Yung-Hsiang Lu
Computer vision often uses highly accurate Convolutional Neural Networks
(CNNs), but these deep learning models are associated with ever-increasing
energy and computation requirements. Producing more energy-efficient CNNs often
requires model training which can be cost-prohibitive. We propose a novel,
automated method to make a pretrained CNN more energy-efficient without
re-training. Given a pretrained CNN, we insert a threshold layer that filters
activations from the preceding layers to identify regions of the image that are
irrelevant, i.e. can be ignored by the following layers while maintaining
accuracy. Our modified focused convolution operation saves inference latency
(by up to 25%) and energy costs (by up to 22%) on various popular pretrained
CNNs, with little to no loss in accuracy.