Temporal Decisions: Leveraging Temporal Correlation for Efficient
  Decisions in Early Exit Neural Networks
Max Sponner Lorenzo Servadei Bernd Waschneck Robert Wille Akash Kumar
Deep Learning is becoming increasingly relevant in Embedded and
Internet-of-things applications. However, deploying models on embedded devices
poses a challenge due to their resource limitations. This can impact the
model's inference accuracy and latency. One potential solution are Early Exit
Neural Networks, which adjust model depth dynamically through additional
classifiers attached between their hidden layers. However, the real-time
termination decision mechanism is critical for the system's efficiency,
latency, and sustained accuracy.
  This paper introduces Difference Detection and Temporal Patience as decision
mechanisms for Early Exit Neural Networks. They leverage the temporal
correlation present in sensor data streams to efficiently terminate the
inference. We evaluate their effectiveness in health monitoring, image
classification, and wake-word detection tasks. Our novel contributions were
able to reduce the computational footprint compared to established decision
mechanisms significantly while maintaining higher accuracy scores. We achieved
a reduction of mean operations per inference by up to 80% while maintaining
accuracy levels within 5% of the original model.
  These findings highlight the importance of considering temporal correlation
in sensor data to improve the termination decision.