On the Effectiveness of LayerNorm Tuning for Continual Learning in
  Vision Transformers
Thomas De Min Massimiliano Mancini Karteek Alahari Xavier Alameda-Pineda Elisa Ricci
State-of-the-art rehearsal-free continual learning methods exploit the
peculiarities of Vision Transformers to learn task-specific prompts,
drastically reducing catastrophic forgetting. However, there is a tradeoff
between the number of learned parameters and the performance, making such
models computationally expensive. In this work, we aim to reduce this cost
while maintaining competitive performance. We achieve this by revisiting and
extending a simple transfer learning idea: learning task-specific normalization
layers. Specifically, we tune the scale and bias parameters of LayerNorm for
each continual learning task, selecting them at inference time based on the
similarity between task-specific keys and the output of the pre-trained model.
To make the classifier robust to incorrect selection of parameters during
inference, we introduce a two-stage training procedure, where we first optimize
the task-specific parameters and then train the classifier with the same
selection procedure of the inference time. Experiments on ImageNet-R and
CIFAR-100 show that our method achieves results that are either superior or on
par with {the state of the art} while being computationally cheaper.