Coupling public and private gradient provably helps optimization
Ruixuan Liu Zhiqi Bu Yu-xiang Wang Sheng Zha George Karypis
The success of large neural networks is crucially determined by the
availability of data. It has been observed that training only on a small amount
of public data, or privately on the abundant private data can lead to
undesirable degradation of accuracy. In this work, we leverage both private and
public data to improve the optimization, by coupling their gradients via a
weighted linear combination. We formulate an optimal solution for the optimal
weight in the convex setting to indicate that the weighting coefficient should
be hyperparameter-dependent. Then, we prove the acceleration in the convergence
of non-convex loss and the effects of hyper-parameters such as privacy budget,
number of iterations, batch size, and model size on the choice of the weighting
coefficient. We support our analysis with empirical experiments across language
and vision benchmarks, and provide a guideline for choosing the optimal weight
of the gradient coupling.