Differentially Private Non-convex Learning for Multi-layer Neural
  Networks
Hanpu Shen Cheng-Long Wang Zihang Xiang Yiming Ying Di Wang
This paper focuses on the problem of Differentially Private Stochastic
Optimization for (multi-layer) fully connected neural networks with a single
output node. In the first part, we examine cases with no hidden nodes,
specifically focusing on Generalized Linear Models (GLMs). We investigate the
well-specific model where the random noise possesses a zero mean, and the link
function is both bounded and Lipschitz continuous. We propose several
algorithms and our analysis demonstrates the feasibility of achieving an excess
population risk that remains invariant to the data dimension. We also delve
into the scenario involving the ReLU link function, and our findings mirror
those of the bounded link function. We conclude this section by contrasting
well-specified and misspecified models, using ReLU regression as a
representative example.
  In the second part of the paper, we extend our ideas to two-layer neural
networks with sigmoid or ReLU activation functions in the well-specified model.
In the third part, we study the theoretical guarantees of DP-SGD in Abadi et
al. (2016) for fully connected multi-layer neural networks. By utilizing recent
advances in Neural Tangent Kernel theory, we provide the first excess
population risk when both the sample size and the width of the network are
sufficiently large. Additionally, we discuss the role of some parameters in
DP-SGD regarding their utility, both theoretically and empirically.