LazyDP: Co-Designing Algorithm-Software for Scalable Training of
  Differentially Private Recommendation Models
Juntaek Lim Youngeun Kwon Ranggi Hwang Kiwan Maeng G. Edward Suh Minsoo Rhu
Differential privacy (DP) is widely being employed in the industry as a
practical standard for privacy protection. While private training of computer
vision or natural language processing applications has been studied
extensively, the computational challenges of training of recommender systems
(RecSys) with DP have not been explored. In this work, we first present our
detailed characterization of private RecSys training using DP-SGD, root-causing
its several performance bottlenecks. Specifically, we identify DP-SGD's noise
sampling and noisy gradient update stage to suffer from a severe compute and
memory bandwidth limitation, respectively, causing significant performance
overhead in training private RecSys. Based on these findings, we propose
LazyDP, an algorithm-software co-design that addresses the compute and memory
challenges of training RecSys with DP-SGD. Compared to a state-of-the-art
DP-SGD training system, we demonstrate that LazyDP provides an average 119x
training throughput improvement while also ensuring mathematically equivalent,
differentially private RecSys models to be trained.