Pre-training Differentially Private Models with Limited Public Data
Zhiqi Bu Xinwei Zhang Mingyi Hong Sheng Zha George Karypis
The superior performance of large foundation models relies on the use of
massive amounts of high-quality data, which often contain sensitive, private
and copyrighted material that requires formal protection. While differential
privacy (DP) is a prominent method to gauge the degree of security provided to
the models, its application is commonly limited to the model fine-tuning stage,
due to the performance degradation when applying DP during the pre-training
stage. Consequently, DP is yet not capable of protecting a substantial portion
of the data used during the initial pre-training process.
  In this work, we first provide a theoretical understanding of the efficacy of
DP training by analyzing the per-iteration loss improvement. We make a key
observation that DP optimizers' performance degradation can be significantly
mitigated by the use of limited public data, which leads to a novel DP
continual pre-training strategy. Empirically, using only 10\% of public data,
our strategy can achieve DP accuracy of 41.5\% on ImageNet-21k (with
$\epsilon=8$), as well as non-DP accuracy of 55.7\% and and 60.0\% on
downstream tasks Places365 and iNaturalist-2021, respectively, on par with
state-of-the-art standard pre-training and substantially outperforming existing
DP pre-trained models.