Hardware-Aware DNN Compression via Diverse Pruning and Mixed-Precision
  Quantization
Konstantinos Balaskas Andreas Karatzas Christos Sad Kostas Siozios Iraklis Anagnostopoulos Georgios Zervakis Jörg Henkel
Deep Neural Networks (DNNs) have shown significant advantages in a wide
variety of domains. However, DNNs are becoming computationally intensive and
energy hungry at an exponential pace, while at the same time, there is a vast
demand for running sophisticated DNN-based services on resource constrained
embedded devices. In this paper, we target energy-efficient inference on
embedded DNN accelerators. To that end, we propose an automated framework to
compress DNNs in a hardware-aware manner by jointly employing pruning and
quantization. We explore, for the first time, per-layer fine- and
coarse-grained pruning, in the same DNN architecture, in addition to low
bit-width mixed-precision quantization for weights and activations.
Reinforcement Learning (RL) is used to explore the associated design space and
identify the pruning-quantization configuration so that the energy consumption
is minimized whilst the prediction accuracy loss is retained at acceptable
levels. Using our novel composite RL agent we are able to extract
energy-efficient solutions without requiring retraining and/or fine tuning. Our
extensive experimental evaluation over widely used DNNs and the CIFAR-10/100
and ImageNet datasets demonstrates that our framework achieves $39\%$ average
energy reduction for $1.7\%$ average accuracy loss and outperforms
significantly the state-of-the-art approaches.