Value-Driven Mixed-Precision Quantization for Patch-Based Inference on
  Microcontrollers
Wei Tao Shenglin He Kai Lu Xiaoyang Qu Guokuan Li Jiguang Wan Jianzong Wang Jing Xiao
Deploying neural networks on microcontroller units (MCUs) presents
substantial challenges due to their constrained computation and memory
resources. Previous researches have explored patch-based inference as a
strategy to conserve memory without sacrificing model accuracy. However, this
technique suffers from severe redundant computation overhead, leading to a
substantial increase in execution latency. A feasible solution to address this
issue is mixed-precision quantization, but it faces the challenges of accuracy
degradation and a time-consuming search time. In this paper, we propose
QuantMCU, a novel patch-based inference method that utilizes value-driven
mixed-precision quantization to reduce redundant computation. We first utilize
value-driven patch classification (VDPC) to maintain the model accuracy. VDPC
classifies patches into two classes based on whether they contain outlier
values. For patches containing outlier values, we apply 8-bit quantization to
the feature maps on the dataflow branches that follow. In addition, for patches
without outlier values, we utilize value-driven quantization search (VDQS) on
the feature maps of their following dataflow branches to reduce search time.
Specifically, VDQS introduces a novel quantization search metric that takes
into account both computation and accuracy, and it employs entropy as an
accuracy representation to avoid additional training. VDQS also adopts an
iterative approach to determine the bitwidth of each feature map to further
accelerate the search process. Experimental results on real-world MCU devices
show that QuantMCU can reduce computation by 2.2x on average while maintaining
comparable model accuracy compared to the state-of-the-art patch-based
inference methods.