Physics-informed RL for Maximal Safety Probability Estimation
Hikaru Hoshino Yorie Nakahira
Accurate risk quantification and reachability analysis are crucial for safe
control and learning, but sampling from rare events, risky states, or long-term
trajectories can be prohibitively costly. Motivated by this, we study how to
estimate the long-term safety probability of maximally safe actions without
sufficient coverage of samples from risky states and long-term trajectories.
The use of maximal safety probability in control and learning is expected to
avoid conservative behaviors due to over-approximation of risk. Here, we first
show that long-term safety probability, which is multiplicative in time, can be
converted into additive costs and be solved using standard reinforcement
learning methods. We then derive this probability as solutions of partial
differential equations (PDEs) and propose Physics-Informed Reinforcement
Learning (PIRL) algorithm. The proposed method can learn using sparse rewards
because the physics constraints help propagate risk information through
neighbors. This suggests that, for the purpose of extracting more information
for efficient learning, physics constraints can serve as an alternative to
reward shaping. The proposed method can also estimate long-term risk using
short-term samples and deduce the risk of unsampled states. This feature is in
stark contrast with the unconstrained deep RL that demands sufficient data
coverage. These merits of the proposed method are demonstrated in numerical
simulation.