FSD-Inference: Fully Serverless Distributed Inference with Scalable
  Cloud Communication
Joe Oakley Hakan Ferhatosmanoglu
Serverless computing offers attractive scalability, elasticity and
cost-effectiveness. However, constraints on memory, CPU and function runtime
have hindered its adoption for data-intensive applications and machine learning
(ML) workloads. Traditional 'server-ful' platforms enable distributed
computation via fast networks and well-established inter-process communication
(IPC) mechanisms such as MPI and shared memory. In the absence of such
solutions in the serverless domain, parallel computation with significant IPC
requirements is challenging. We present FSD-Inference, the first fully
serverless and highly scalable system for distributed ML inference. We explore
potential communication channels, in conjunction with Function-as-a-Service
(FaaS) compute, to design a state-of-the-art solution for distributed ML within
the context of serverless data-intensive computing. We introduce novel fully
serverless communication schemes for ML inference workloads, leveraging both
cloud-based publish-subscribe/queueing and object storage offerings. We
demonstrate how publish-subscribe/queueing services can be adapted for FaaS IPC
with comparable performance to object storage, while offering significantly
reduced cost at high parallelism levels. We conduct in-depth experiments on
benchmark DNNs of various sizes. The results show that when compared to
server-based alternatives, FSD-Inference is significantly more cost-effective
and scalable, and can even achieve competitive performance against optimized
HPC solutions. Experiments also confirm that our serverless solution can handle
large distributed workloads and leverage high degrees of FaaS parallelism.