Distributed Continual Learning with CoCoA in High-dimensional Linear
  Regression
Martin Hellkvist Ayça Özçelikkale Anders Ahlén
We consider estimation under scenarios where the signals of interest exhibit
change of characteristics over time. In particular, we consider the continual
learning problem where different tasks, e.g., data with different
distributions, arrive sequentially and the aim is to perform well on the newly
arrived task without performance degradation on the previously seen tasks. In
contrast to the continual learning literature focusing on the centralized
setting, we investigate the problem from a distributed estimation perspective.
We consider the well-established distributed learning algorithm COCOA, which
distributes the model parameters and the corresponding features over the
network. We provide exact analytical characterization for the generalization
error of COCOA under continual learning for linear regression in a range of
scenarios, where overparameterization is of particular interest. These
analytical results characterize how the generalization error depends on the
network structure, the task similarity and the number of tasks, and show how
these dependencies are intertwined. In particular, our results show that the
generalization error can be significantly reduced by adjusting the network
size, where the most favorable network size depends on task similarity and the
number of tasks. We present numerical results verifying the theoretical
analysis and illustrate the continual learning performance of COCOA with a
digit classification task.