Frobenius-Type Norms and Inner Products of Matrices and Linear Maps with
  Applications to Neural Network Training
Roland Herzog Frederik Köhne Leonie Kreis Anton Schiela
The Frobenius norm is a frequent choice of norm for matrices. In particular,
the underlying Frobenius inner product is typically used to evaluate the
gradient of an objective with respect to matrix variable, such as those
occuring in the training of neural networks. We provide a broader view on the
Frobenius norm and inner product for linear maps or matrices, and establish
their dependence on inner products in the domain and co-domain spaces. This
shows that the classical Frobenius norm is merely one special element of a
family of more general Frobenius-type norms. The significant extra freedom
furnished by this realization can be used, among other things, to precondition
neural network training.