Batch Clipping and Adaptive Layerwise Clipping for Differential Private
  Stochastic Gradient Descent
Toan N. Nguyen Phuong Ha Nguyen Lam M. Nguyen Marten Van Dijk
Each round in Differential Private Stochastic Gradient Descent (DPSGD)
transmits a sum of clipped gradients obfuscated with Gaussian noise to a
central server which uses this to update a global model which often represents
a deep neural network. Since the clipped gradients are computed separately,
which we call Individual Clipping (IC), deep neural networks like resnet-18
cannot use Batch Normalization Layers (BNL) which is a crucial component in
deep neural networks for achieving a high accuracy. To utilize BNL, we
introduce Batch Clipping (BC) where, instead of clipping single gradients as in
the orginal DPSGD, we average and clip batches of gradients. Moreover, the
model entries of different layers have different sensitivities to the added
Gaussian noise. Therefore, Adaptive Layerwise Clipping methods (ALC), where
each layer has its own adaptively finetuned clipping constant, have been
introduced and studied, but so far without rigorous DP proofs. In this paper,
we propose {\em a new ALC and provide rigorous DP proofs for both BC and ALC}.
Experiments show that our modified DPSGD with BC and ALC for CIFAR-$10$ with
resnet-$18$ converges while DPSGD with IC and ALC does not.