Sparse-DySta: Sparsity-Aware Dynamic and Static Scheduling for Sparse
  Multi-DNN Workloads
Hongxiang Fan Stylianos I. Venieris Alexandros Kouris Nicholas D. Lane
Running multiple deep neural networks (DNNs) in parallel has become an
emerging workload in both edge devices, such as mobile phones where multiple
tasks serve a single user for daily activities, and data centers, where various
requests are raised from millions of users, as seen with large language models.
To reduce the costly computational and memory requirements of these workloads,
various efficient sparsification approaches have been introduced, resulting in
widespread sparsity across different types of DNN models. In this context,
there is an emerging need for scheduling sparse multi-DNN workloads, a problem
that is largely unexplored in previous literature. This paper systematically
analyses the use-cases of multiple sparse DNNs and investigates the
opportunities for optimizations. Based on these findings, we propose Dysta, a
novel bi-level dynamic and static scheduler that utilizes both static sparsity
patterns and dynamic sparsity information for the sparse multi-DNN scheduling.
Both static and dynamic components of Dysta are jointly designed at the
software and hardware levels, respectively, to improve and refine the
scheduling approach. To facilitate future progress in the study of this class
of workloads, we construct a public benchmark that contains sparse multi-DNN
workloads across different deployment scenarios, spanning from mobile phones
and AR/VR wearables to data centers. A comprehensive evaluation on the sparse
multi-DNN benchmark demonstrates that our proposed approach outperforms the
state-of-the-art methods with up to 10% decrease in latency constraint
violation rate and nearly 4X reduction in average normalized turnaround time.
Our artifacts and code are publicly available at:
https://github.com/SamsungLabs/Sparse-Multi-DNN-Scheduling.