Stochastic Modified Equations and Dynamics of Dropout Algorithm
Zhongwang Zhang Yuqing Li Tao Luo Zhi-Qin John Xu
Dropout is a widely utilized regularization technique in the training of
neural networks, nevertheless, its underlying mechanism and its impact on
achieving good generalization abilities remain poorly understood. In this work,
we derive the stochastic modified equations for analyzing the dynamics of
dropout, where its discrete iteration process is approximated by a class of
stochastic differential equations. In order to investigate the underlying
mechanism by which dropout facilitates the identification of flatter minima, we
study the noise structure of the derived stochastic modified equation for
dropout. By drawing upon the structural resemblance between the Hessian and
covariance through several intuitive approximations, we empirically demonstrate
the universal presence of the inverse variance-flatness relation and the
Hessian-variance relation, throughout the training process of dropout. These
theoretical and empirical findings make a substantial contribution to our
understanding of the inherent tendency of dropout to locate flatter minima.