Efficient Post-Training Augmentation for Adaptive Inference in
  Heterogeneous and Distributed IoT Environments
Max Sponner Lorenzo Servadei Bernd Waschneck Robert Wille Akash Kumar
Early Exit Neural Networks (EENNs) present a solution to enhance the
efficiency of neural network deployments. However, creating EENNs is
challenging and requires specialized domain knowledge, due to the large amount
of additional design choices. To address this issue, we propose an automated
augmentation flow that focuses on converting an existing model into an EENN. It
performs all required design decisions for the deployment to heterogeneous or
distributed hardware targets: Our framework constructs the EENN architecture,
maps its subgraphs to the hardware targets, and configures its decision
mechanism. To the best of our knowledge, it is the first framework that is able
to perform all of these steps.
  We evaluated our approach on a collection of Internet-of-Things and standard
image classification use cases. For a speech command detection task, our
solution was able to reduce the mean operations per inference by 59.67%. For an
ECG classification task, it was able to terminate all samples early, reducing
the mean inference energy by 74.9% and computations by 78.3%. On CIFAR-10, our
solution was able to achieve up to a 58.75% reduction in computations.
  The search on a ResNet-152 base model for CIFAR-10 took less than nine hours
on a laptop CPU. Our proposed approach enables the creation of EENN optimized
for IoT environments and can reduce the inference cost of Deep Learning
applications on embedded and fog platforms, while also significantly reducing
the search cost - making it more accessible for scientists and engineers in
industry and research. The low search cost improves the accessibility of EENNs,
with the potential to improve the efficiency of neural networks in a wide range
of practical applications.