Provable Traffic Rule Compliance in Safe Reinforcement Learning on the
  Open Sea
Hanna Krasowski Matthias Althoff
Autonomous vehicles have to obey traffic rules. These rules are often
formalized using temporal logic, resulting in constraints that are hard to
solve using optimization-based motion planners. Reinforcement Learning (RL) is
a promising method to find motion plans adhering to temporal logic
specifications. However, vanilla RL algorithms are based on random exploration,
which is inherently unsafe. To address this issue, we propose a provably safe
RL approach that always complies with traffic rules. As a specific application
area, we consider vessels on the open sea, which must adhere to the Convention
on the International Regulations for Preventing Collisions at Sea (COLREGS). We
introduce an efficient verification approach that determines the compliance of
actions with respect to the COLREGS formalized using temporal logic. Our action
verification is integrated into the RL process so that the agent only selects
verified actions. In contrast to agents that only integrate the traffic rule
information in the reward function, our provably safe agent always complies
with the formalized rules in critical maritime traffic situations and, thus,
never causes a collision.