An Intentional Forgetting-Driven Self-Healing Method For Deep
  Reinforcement Learning Systems
Ahmed Haj Yahmed Rached Bouchoucha Houssem Ben Braiek Foutse Khomh
Deep reinforcement learning (DRL) is increasingly applied in large-scale
productions like Netflix and Facebook. As with most data-driven systems, DRL
systems can exhibit undesirable behaviors due to environmental drifts, which
often occur in constantly-changing production settings. Continual Learning (CL)
is the inherent self-healing approach for adapting the DRL agent in response to
the environment's conditions shifts. However, successive shifts of considerable
magnitude may cause the production environment to drift from its original
state. Recent studies have shown that these environmental drifts tend to drive
CL into long, or even unsuccessful, healing cycles, which arise from
inefficiencies such as catastrophic forgetting, warm-starting failure, and slow
convergence. In this paper, we propose Dr. DRL, an effective self-healing
approach for DRL systems that integrates a novel mechanism of intentional
forgetting into vanilla CL to overcome its main issues. Dr. DRL deliberately
erases the DRL system's minor behaviors to systematically prioritize the
adaptation of the key problem-solving skills. Using well-established DRL
algorithms, Dr. DRL is compared with vanilla CL on various drifted
environments. Dr. DRL is able to reduce, on average, the healing time and
fine-tuning episodes by, respectively, 18.74% and 17.72%. Dr. DRL successfully
helps agents to adapt to 19.63% of drifted environments left unsolved by
vanilla CL while maintaining and even enhancing by up to 45% the obtained
rewards for drifted environments that are resolved by both approaches.