Deep Reinforcement Learning based Online Scheduling Policy for Deep
  Neural Network Multi-Tenant Multi-Accelerator Systems
Francesco G. Blanco Enrico Russo Maurizio Palesi Davide Patti Giuseppe Ascia Vincenzo Catania
Currently, there is a growing trend of outsourcing the execution of DNNs to
cloud services. For service providers, managing multi-tenancy and ensuring
high-quality service delivery, particularly in meeting stringent execution time
constraints, assumes paramount importance, all while endeavoring to maintain
cost-effectiveness. In this context, the utilization of heterogeneous
multi-accelerator systems becomes increasingly relevant. This paper presents
RELMAS, a low-overhead deep reinforcement learning algorithm designed for the
online scheduling of DNNs in multi-tenant environments, taking into account the
dataflow heterogeneity of accelerators and memory bandwidths contentions. By
doing so, service providers can employ the most efficient scheduling policy for
user requests, optimizing Service-Level-Agreement (SLA) satisfaction rates and
enhancing hardware utilization. The application of RELMAS to a heterogeneous
multi-accelerator system composed of various instances of Simba and Eyeriss
sub-accelerators resulted in up to a 173% improvement in SLA satisfaction rate
compared to state-of-the-art scheduling techniques across different workload
scenarios, with less than a 1.5% energy overhead.