Continual Learning with Low Rank Adaptation
Martin Wistuba Prabhu Teja Sivaprasad Lukas Balles Giovanni Zappella
Recent work using pretrained transformers has shown impressive performance
when fine-tuned with data from the downstream problem of interest. However,
they struggle to retain that performance when the data characteristics changes.
In this paper, we focus on continual learning, where a pre-trained transformer
is updated to perform well on new data, while retaining its performance on data
it was previously trained on. Earlier works have tackled this primarily through
methods inspired from prompt tuning. We question this choice, and investigate
the applicability of Low Rank Adaptation (LoRA) to continual learning. On a
range of domain-incremental learning benchmarks, our LoRA-based solution,
CoLoR, yields state-of-the-art performance, while still being as parameter
efficient as the prompt tuning based methods.