Distal Interference: Exploring the Limits of Model-Based Continual
  Learning
Heinrich van Deventer Anna Sergeevna Bosman
Continual learning is the sequential learning of different tasks by a machine
learning model. Continual learning is known to be hindered by catastrophic
interference or forgetting, i.e. rapid unlearning of earlier learned tasks when
new tasks are learned. Despite their practical success, artificial neural
networks (ANNs) are prone to catastrophic interference. This study analyses how
gradient descent and overlapping representations between distant input points
lead to distal interference and catastrophic interference. Distal interference
refers to the phenomenon where training a model on a subset of the domain leads
to non-local changes on other subsets of the domain. This study shows that
uniformly trainable models without distal interference must be exponentially
large. A novel antisymmetric bounded exponential layer B-spline ANN
architecture named ABEL-Spline is proposed that can approximate any continuous
function, is uniformly trainable, has polynomial computational complexity, and
provides some guarantees for distal interference. Experiments are presented to
demonstrate the theoretical properties of ABEL-Splines. ABEL-Splines are also
evaluated on benchmark regression problems. It is concluded that the weaker
distal interference guarantees in ABEL-Splines are insufficient for model-only
continual learning. It is conjectured that continual learning with polynomial
complexity models requires augmentation of the training data or algorithm.