Hyperparameters in Continual Learning: a Reality Check
Sungmin Cha Kyunghyun Cho
Various algorithms for continual learning (CL) have been designed with the
goal of effectively alleviating the trade-off between stability and plasticity
during the CL process. To achieve this goal, tuning appropriate hyperparameters
for each algorithm is essential. As an evaluation protocol, it has been common
practice to train a CL algorithm using diverse hyperparameter values on a CL
scenario constructed with a benchmark dataset. Subsequently, the best
performance attained with the optimal hyperparameter value serves as the
criterion for evaluating the CL algorithm. In this paper, we contend that this
evaluation protocol is not only impractical but also incapable of effectively
assessing the CL capability of a CL algorithm. Returning to the fundamental
principles of model evaluation in machine learning, we propose an evaluation
protocol that involves Hyperparameter Tuning and Evaluation phases. Those
phases consist of different datasets but share the same CL scenario. In the
Hyperparameter Tuning phase, each algorithm is iteratively trained with
different hyperparameter values to find the optimal hyperparameter values.
Subsequently, in the Evaluation phase, the optimal hyperparameter values is
directly applied for training each algorithm, and their performance in the
Evaluation phase serves as the criterion for evaluating them. Through
experiments on CIFAR-100 and ImageNet-100 based on the proposed protocol in
class-incremental learning, we not only observed that the existing evaluation
method fail to properly assess the CL capability of each algorithm but also
observe that some recently proposed state-of-the-art algorithms, which reported
superior performance, actually exhibit inferior performance compared to the
previous algorithm.