Combined Scheduling, Memory Allocation and Tensor Replacement for
  Minimizing Off-Chip Data Accesses of DNN Accelerators
Yi Li Aarti Gupta Sharad Malik
Specialized hardware accelerators have been extensively used for Deep Neural
Networks (DNNs) to provide power/performance benefits. These accelerators
contain specialized hardware that supports DNN operators, and scratchpad memory
for storing the tensor operands. Often, the size of the scratchpad is
insufficient to store all the tensors needed for the computation, and
additional data accesses are needed to move tensors back and forth from host
memory during the computation with significant power/performance overhead. The
volume of these additional data accesses depends on the operator schedule, and
memory allocation (specific locations selected for the tensors in the
scratchpad). We propose an optimization framework, named COSMA, for mapping
DNNs to an accelerator that finds the optimal operator schedule, memory
allocation and tensor replacement that minimizes the additional data accesses.
COSMA provides an Integer Linear Programming (ILP) formulation to generate the
optimal solution for mapping a DNN to the accelerator for a given scratchpad
size. We demonstrate that, using an off-the-shelf ILP solver, COSMA obtains the
optimal solution in seconds for a wide-range of state-of-the-art DNNs for
different applications. Further, it out-performs existing methods by reducing
on average 84% of the non-compulsory data accesses. We further propose a
divide-and-conquer heuristic to scale up to certain complex DNNs generated by
Neural Architecture Search, and this heuristic solution reduces on average 85%
data accesses compared with other works.