On the Convergence of Differentially-Private Fine-tuning: To Linearly
  Probe or to Fully Fine-tune?
Shuqi Ke Charlie Hou Giulia Fanti Sewoong Oh
Differentially private (DP) machine learning pipelines typically involve a
two-phase process: non-private pre-training on a public dataset, followed by
fine-tuning on private data using DP optimization techniques. In the DP
setting, it has been observed that full fine-tuning may not always yield the
best test accuracy, even for in-distribution data. This paper (1) analyzes the
training dynamics of DP linear probing (LP) and full fine-tuning (FT), and (2)
explores the phenomenon of sequential fine-tuning, starting with linear probing
and transitioning to full fine-tuning (LP-FT), and its impact on test loss. We
provide theoretical insights into the convergence of DP fine-tuning within an
overparameterized neural network and establish a utility curve that determines
the allocation of privacy budget between linear probing and full fine-tuning.
The theoretical results are supported by empirical evaluations on various
benchmarks and models. The findings reveal the complex nature of DP fine-tuning
methods. These results contribute to a deeper understanding of DP machine
learning and highlight the importance of considering the allocation of privacy
budget in the fine-tuning process.