Revisiting Supervision for Continual Representation Learning
Daniel Marczak Sebastian Cygert Tomasz Trzciński Bartłomiej Twardowski
In the field of continual learning, models are designed to learn tasks one
after the other. While most research has centered on supervised continual
learning, recent studies have highlighted the strengths of self-supervised
continual representation learning. The improved transferability of
representations built with self-supervised methods is often associated with the
role played by the multi-layer perceptron projector. In this work, we depart
from this observation and reexamine the role of supervision in continual
representation learning. We reckon that additional information, such as human
annotations, should not deteriorate the quality of representations. Our
findings show that supervised models when enhanced with a multi-layer
perceptron head, can outperform self-supervised models in continual
representation learning.