OmniBoost: Boosting Throughput of Heterogeneous Embedded Devices under
  Multi-DNN Workload
Andreas Karatzas Iraklis Anagnostopoulos
Modern Deep Neural Networks (DNNs) exhibit profound efficiency and accuracy
properties. This has introduced application workloads that comprise of multiple
DNN applications, raising new challenges regarding workload distribution.
Equipped with a diverse set of accelerators, newer embedded system present
architectural heterogeneity, which current run-time controllers are unable to
fully utilize. To enable high throughput in multi-DNN workloads, such a
controller is ought to explore hundreds of thousands of possible solutions to
exploit the underlying heterogeneity. In this paper, we propose OmniBoost, a
lightweight and extensible multi-DNN manager for heterogeneous embedded
devices. We leverage stochastic space exploration and we combine it with a
highly accurate performance estimator to observe a x4.6 average throughput
boost compared to other state-of-the-art methods. The evaluation was performed
on the HiKey970 development board.