Collaborative Inference via Dynamic Composition of Tiny AI Accelerators
  on MCUs
Taesik Gong Si Young Jang Utku Günay Acer Fahim Kawsar Chulhong Min
The advent of tiny AI accelerators opens opportunities for deep neural
network deployment at the extreme edge, offering reduced latency, lower power
cost, and improved privacy in on-device ML inference. Despite these
advancements, challenges persist due to inherent limitations of these
accelerators, such as restricted onboard memory and single-device focus. This
paper introduces Synergy, a system that dynamically composes tiny AI
accelerators for multi-tenant models, effectively addressing tinyML's critical
challenges for the increasing demand for on-device AI. A key feature of Synergy
is its virtual computing space, providing a unified, virtualized view of
resources and enabling efficient task mapping to physical devices. Synergy's
runtime orchestration module ensures optimal inference across dynamic and
heterogeneous accelerators. Our evaluations with 7 baselines and 8 models
demonstrate that Synergy improves throughput by an average of 8.0X compared to
baselines.