Diagnosing Catastrophe: Large parts of accuracy loss in continual
  learning can be accounted for by readout misalignment
Daniel Anthes Sushrut Thorat Peter König Tim C. Kietzmann
Unlike primates, training artificial neural networks on changing data
distributions leads to a rapid decrease in performance on old tasks. This
phenomenon is commonly referred to as catastrophic forgetting. In this paper,
we investigate the representational changes that underlie this performance
decrease and identify three distinct processes that together account for the
phenomenon. The largest component is a misalignment between hidden
representations and readout layers. Misalignment occurs due to learning on
additional tasks and causes internal representations to shift. Representational
geometry is partially conserved under this misalignment and only a small part
of the information is irrecoverably lost. All types of representational changes
scale with the dimensionality of hidden representations. These insights have
implications for deep learning applications that need to be continuously
updated, but may also aid aligning ANN models to the rather robust biological
vision.