Tuning for the Unknown: Revisiting Evaluation Strategies for Lifelong RL
Golnaz Mesbahi Olya Mastikhina Parham Mohammad Panahi Martha White Adam White
In continual or lifelong reinforcement learning access to the environment
should be limited. If we aspire to design algorithms that can run for
long-periods of time, continually adapting to new, unexpected situations then
we must be willing to deploy our agents without tuning their hyperparameters
over the agent's entire lifetime. The standard practice in deep RL -- and even
continual RL -- is to assume unfettered access to deployment environment for
the full lifetime of the agent. This paper explores the notion that progress in
lifelong RL research has been held back by inappropriate empirical
methodologies. In this paper we propose a new approach for tuning and
evaluating lifelong RL agents where only one percent of the experiment data can
be used for hyperparameter tuning. We then conduct an empirical study of DQN
and Soft Actor Critic across a variety of continuing and non-stationary
domains. We find both methods generally perform poorly when restricted to
one-percent tuning, whereas several algorithmic mitigations designed to
maintain network plasticity perform surprising well. In addition, we find that
properties designed to measure the network's ability to learn continually
indeed correlate with performance under one-percent tuning.