Mobile-Cloud Inference for Collaborative Intelligence
Mateen Ulhaq
As AI applications for mobile devices become more prevalent, there is an
increasing need for faster execution and lower energy consumption for deep
learning model inference. Historically, the models run on mobile devices have
been smaller and simpler in comparison to large state-of-the-art research
models, which can only run on the cloud. However, cloud-only inference has
drawbacks such as increased network bandwidth consumption and higher latency.
In addition, cloud-only inference requires the input data (images, audio) to be
fully transferred to the cloud, creating concerns about potential privacy
breaches.
  There is an alternative approach: shared mobile-cloud inference. Partial
inference is performed on the mobile in order to reduce the dimensionality of
the input data and arrive at a compact feature tensor, which is a latent space
representation of the input signal. The feature tensor is then transmitted to
the server for further inference. This strategy can reduce inference latency,
energy consumption, and network bandwidth usage, as well as provide privacy
protection, because the original signal never leaves the mobile. Further
performance gain can be achieved by compressing the feature tensor before its
transmission.