Safety Margins for Reinforcement Learning
Alexander Grushin Walt Woods Alvaro Velasquez Simon Khan
Any autonomous controller will be unsafe in some situations. The ability to
quantitatively identify when these unsafe situations are about to occur is
crucial for drawing timely human oversight in, e.g., freight transportation
applications. In this work, we demonstrate that the true criticality of an
agent's situation can be robustly defined as the mean reduction in reward given
some number of random actions. Proxy criticality metrics that are computable in
real-time (i.e., without actually simulating the effects of random actions) can
be compared to the true criticality, and we show how to leverage these proxy
metrics to generate safety margins, which directly tie the consequences of
potentially incorrect actions to an anticipated loss in overall performance. We
evaluate our approach on learned policies from APE-X and A3C within an Atari
environment, and demonstrate how safety margins decrease as agents approach
failure states. The integration of safety margins into programs for monitoring
deployed agents allows for the real-time identification of potentially
catastrophic situations.