Deep Policy Optimization with Temporal Logic Constraints
Ameesh Shah Cameron Voloshin Chenxi Yang Abhinav Verma Swarat Chaudhuri Sanjit A. Seshia
Temporal logics, such as linear temporal logic (LTL), offer a precise means
of specifying tasks for (deep) reinforcement learning (RL) agents. In our work,
we consider the setting where the task is specified by an LTL objective and
there is an additional scalar reward that we need to optimize. Previous works
focus either on learning a LTL task-satisfying policy alone or are restricted
to finite state spaces. We make two contributions: First, we introduce an
RL-friendly approach to this setting by formulating this problem as a single
optimization objective. Our formulation guarantees that an optimal policy will
be reward-maximal from the set of policies that maximize the likelihood of
satisfying the LTL specification. Second, we address a sparsity issue that
often arises for LTL-guided Deep RL policies by introducing Cycle Experience
Replay (CyclER), a technique that automatically guides RL agents towards the
satisfaction of an LTL specification. Our experiments demonstrate the efficacy
of CyclER in finding performant deep RL policies in both continuous and
discrete experimental domains.