Combining Explicit and Implicit Regularization for Efficient Learning in
  Deep Networks
Dan Zhao
Works on implicit regularization have studied gradient trajectories during
the optimization process to explain why deep networks favor certain kinds of
solutions over others. In deep linear networks, it has been shown that gradient
descent implicitly regularizes toward low-rank solutions on matrix
completion/factorization tasks. Adding depth not only improves performance on
these tasks but also acts as an accelerative pre-conditioning that further
enhances this bias towards low-rankedness. Inspired by this, we propose an
explicit penalty to mirror this implicit bias which only takes effect with
certain adaptive gradient optimizers (e.g. Adam). This combination can enable a
degenerate single-layer network to achieve low-rank approximations with
generalization error comparable to deep linear networks, making depth no longer
necessary for learning. The single-layer network also performs competitively or
out-performs various approaches for matrix completion over a range of parameter
and data regimes despite its simplicity. Together with an optimizer's inductive
bias, our findings suggest that explicit regularization can play a role in
designing different, desirable forms of regularization and that a more nuanced
understanding of this interplay may be necessary.