Control invariant set enhanced safe reinforcement learning: improved
  sampling efficiency, guaranteed stability and robustness
Song Bo Bernard T. Agyeman Xunyuan Yin Jinfeng Liu
Reinforcement learning (RL) is an area of significant research interest, and
safe RL in particular is attracting attention due to its ability to handle
safety-driven constraints that are crucial for real-world applications. This
work proposes a novel approach to RL training, called control invariant set
(CIS) enhanced RL, which leverages the advantages of utilizing the explicit
form of CIS to improve stability guarantees and sampling efficiency.
Furthermore, the robustness of the proposed approach is investigated in the
presence of uncertainty. The approach consists of two learning stages: offline
and online. In the offline stage, CIS is incorporated into the reward design,
initial state sampling, and state reset procedures. This incorporation of CIS
facilitates improved sampling efficiency during the offline training process.
In the online stage, RL is retrained whenever the predicted next step state is
outside of the CIS, which serves as a stability criterion, by introducing a
Safety Supervisor to examine the safety of the action and make necessary
corrections. The stability analysis is conducted for both cases, with and
without uncertainty. To evaluate the proposed approach, we apply it to a
simulated chemical reactor. The results show a significant improvement in
sampling efficiency during offline training and closed-loop stability guarantee
in the online implementation, with and without uncertainty.