Dropout Regularization in Extended Generalized Linear Models based on
  Double Exponential Families
Benedikt Lütke Schwienhorst Lucas Kock David J. Nott Nadja Klein
Even though dropout is a popular regularization technique, its theoretical
properties are not fully understood. In this paper we study dropout
regularization in extended generalized linear models based on double
exponential families, for which the dispersion parameter can vary with the
features. A theoretical analysis shows that dropout regularization prefers rare
but important features in both the mean and dispersion, generalizing an earlier
result for conventional generalized linear models. Training is performed using
stochastic gradient descent with adaptive learning rate. To illustrate, we
apply dropout to adaptive smoothing with B-splines, where both the mean and
dispersion parameters are modelled flexibly. The important B-spline basis
functions can be thought of as rare features, and we confirm in experiments
that dropout is an effective form of regularization for mean and dispersion
parameters that improves on a penalized maximum likelihood approach with an
explicit smoothness penalty.