Deep Neural Network Models Trained With A Fixed Random Classifier
  Transfer Better Across Domains
Hafiz Tiomoko Ali Umberto Michieli Ji Joong Moon Daehyun Kim Mete Ozay
The recently discovered Neural collapse (NC) phenomenon states that the
last-layer weights of Deep Neural Networks (DNN), converge to the so-called
Equiangular Tight Frame (ETF) simplex, at the terminal phase of their training.
This ETF geometry is equivalent to vanishing within-class variability of the
last layer activations. Inspired by NC properties, we explore in this paper the
transferability of DNN models trained with their last layer weight fixed
according to ETF. This enforces class separation by eliminating class
covariance information, effectively providing implicit regularization. We show
that DNN models trained with such a fixed classifier significantly improve
transfer performance, particularly on out-of-domain datasets. On a broad range
of fine-grained image classification datasets, our approach outperforms i)
baseline methods that do not perform any covariance regularization (up to 22%),
as well as ii) methods that explicitly whiten covariance of activations
throughout training (up to 19%). Our findings suggest that DNNs trained with
fixed ETF classifiers offer a powerful mechanism for improving transfer
learning across domains.