Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in
  ML Serving
Yinwei Dai Rui Pan Anand Iyer Kai Li Ravi Netravali
Machine learning (ML) inference platforms are tasked with balancing two
competing goals: ensuring high throughput given many requests, and delivering
low-latency responses to support interactive applications. Unfortunately,
existing platform knobs (e.g., batch sizes) fail to ease this fundamental
tension, and instead only enable users to harshly trade off one property for
the other. This paper explores an alternate strategy to taming
throughput-latency tradeoffs by changing the granularity at which inference is
performed. We present Apparate, a system that automatically applies and manages
early exits (EEs) in ML models, whereby certain inputs can exit with results at
intermediate layers. To cope with the time-varying overhead and accuracy
challenges that EEs bring, Apparate repurposes exits to provide continual
feedback that powers several novel runtime monitoring and adaptation
strategies. Apparate lowers median response latencies by 40.5-91.5% and
10.0-24.2% for diverse CV and NLP workloads, respectively, without affecting
throughputs or violating tight accuracy constraints.