Safety-Aware Task Composition for Discrete and Continuous Reinforcement
  Learning
Kevin Leahy Makai Mann Zachary Serlin
Compositionality is a critical aspect of scalable system design.
Reinforcement learning (RL) has recently shown substantial success in task
learning, but has only recently begun to truly leverage composition. In this
paper, we focus on Boolean composition of learned tasks as opposed to
functional or sequential composition. Existing Boolean composition for RL
focuses on reaching a satisfying absorbing state in environments with discrete
action spaces, but does not support composable safety (i.e., avoidance)
constraints. We advance the state of the art in Boolean composition of learned
tasks with three contributions: i) introduce two distinct notions of safety in
this framework; ii) show how to enforce either safety semantics, prove
correctness (under some assumptions), and analyze the trade-offs between the
two safety notions; and iii) extend Boolean composition from discrete action
spaces to continuous action spaces. We demonstrate these techniques using
modified versions of value iteration in a grid world, Deep Q-Network (DQN) in a
grid world with image observations, and Twin Delayed DDPG (TD3) in a
continuous-observation and continuous-action Bullet physics environment. We
believe that these contributions advance the theory of safe reinforcement
learning by allowing zero-shot composition of policies satisfying safety
properties.