AutoReP: Automatic ReLU Replacement for Fast Private Network Inference
Hongwu Peng Shaoyi Huang Tong Zhou Yukui Luo Chenghong Wang Zigeng Wang Jiahui Zhao Xi Xie Ang Li Tony Geng Kaleel Mahmood Wujie Wen Xiaolin Xu Caiwen Ding
The growth of the Machine-Learning-As-A-Service (MLaaS) market has
highlighted clients' data privacy and security issues. Private inference (PI)
techniques using cryptographic primitives offer a solution but often have high
computation and communication costs, particularly with non-linear operators
like ReLU. Many attempts to reduce ReLU operations exist, but they may need
heuristic threshold selection or cause substantial accuracy loss. This work
introduces AutoReP, a gradient-based approach to lessen non-linear operators
and alleviate these issues. It automates the selection of ReLU and polynomial
functions to speed up PI applications and introduces distribution-aware
polynomial approximation (DaPa) to maintain model expressivity while accurately
approximating ReLUs. Our experimental results demonstrate significant accuracy
improvements of 6.12% (94.31%, 12.9K ReLU budget, CIFAR-10), 8.39% (74.92%,
12.9K ReLU budget, CIFAR-100), and 9.45% (63.69%, 55K ReLU budget,
Tiny-ImageNet) over current state-of-the-art methods, e.g., SNL. Morever,
AutoReP is applied to EfficientNet-B2 on ImageNet dataset, and achieved 75.55%
accuracy with 176.1 times ReLU budget reduction.