Using Curiosity for an Even Representation of Tasks in Continual Offline
  Reinforcement Learning
Pankayaraj Pathmanathan Natalia Díaz-Rodríguez Javier Del Ser
In this work, we investigate the means of using curiosity on replay buffers
to improve offline multi-task continual reinforcement learning when tasks,
which are defined by the non-stationarity in the environment, are non labeled
and not evenly exposed to the learner in time. In particular, we investigate
the use of curiosity both as a tool for task boundary detection and as a
priority metric when it comes to retaining old transition tuples, which we
respectively use to propose two different buffers. Firstly, we propose a Hybrid
Reservoir Buffer with Task Separation (HRBTS), where curiosity is used to
detect task boundaries that are not known due to the task agnostic nature of
the problem. Secondly, by using curiosity as a priority metric when it comes to
retaining old transition tuples, a Hybrid Curious Buffer (HCB) is proposed. We
ultimately show that these buffers, in conjunction with regular reinforcement
learning algorithms, can be used to alleviate the catastrophic forgetting issue
suffered by the state of the art on replay buffers when the agent's exposure to
tasks is not equal along time. We evaluate catastrophic forgetting and the
efficiency of our proposed buffers against the latest works such as the Hybrid
Reservoir Buffer (HRB) and the Multi-Time Scale Replay Buffer (MTR) in three
different continual reinforcement learning settings. Experiments were done on
classical control tasks and Metaworld environment. Experiments show that our
proposed replay buffers display better immunity to catastrophic forgetting
compared to existing works in most of the settings.