Jointly Exploring Client Drift and Catastrophic Forgetting in Dynamic
  Learning
Niklas Babendererde Moritz Fuchs Camila Gonzalez Yuri Tolkach Anirban Mukhopadhyay
Federated and Continual Learning have emerged as potential paradigms for the
robust and privacy-aware use of Deep Learning in dynamic environments. However,
Client Drift and Catastrophic Forgetting are fundamental obstacles to
guaranteeing consistent performance. Existing work only addresses these
problems separately, which neglects the fact that the root cause behind both
forms of performance deterioration is connected. We propose a unified analysis
framework for building a controlled test environment for Client Drift -- by
perturbing a defined ratio of clients -- and Catastrophic Forgetting -- by
shifting all clients with a particular strength. Our framework further
leverages this new combined analysis by generating a 3D landscape of the
combined performance impact from both. We demonstrate that the performance drop
through Client Drift, caused by a certain share of shifted clients, is
correlated to the drop from Catastrophic Forgetting resulting from a
corresponding shift strength. Correlation tests between both problems for
Computer Vision (CelebA) and Medical Imaging (PESO) support this new
perspective, with an average Pearson rank correlation coefficient of over 0.94.
Our framework's novel ability of combined spatio-temporal shift analysis allows
us to investigate how both forms of distribution shift behave in mixed
scenarios, opening a new pathway for better generalization. We show that a
combination of moderate Client Drift and Catastrophic Forgetting can even
improve the performance of the resulting model (causing a "Generalization
Bump") compared to when only one of the shifts occurs individually. We apply a
simple and commonly used method from Continual Learning in the federated
setting and observe this phenomenon to be reoccurring, leveraging the ability
of our framework to analyze existing and novel methods for Federated and
Continual Learning.