Assume-Guarantee Reinforcement Learning
Milad Kazemi Mateo Perez Fabio Somenzi Sadegh Soudjani Ashutosh Trivedi Alvaro Velasquez
We present a modular approach to \emph{reinforcement learning} (RL) in
environments consisting of simpler components evolving in parallel. A
monolithic view of such modular environments may be prohibitively large to
learn, or may require unrealizable communication between the components in the
form of a centralized controller. Our proposed approach is based on the
assume-guarantee paradigm where the optimal control for the individual
components is synthesized in isolation by making \emph{assumptions} about the
behaviors of neighboring components, and providing \emph{guarantees} about
their own behavior. We express these \emph{assume-guarantee contracts} as
regular languages and provide automatic translations to scalar rewards to be
used in RL. By combining local probabilities of satisfaction for each
component, we provide a lower bound on the probability of satisfaction of the
complete system. By solving a Markov game for each component, RL can produce a
controller for each component that maximizes this lower bound. The controller
utilizes the information it receives through communication, observations, and
any knowledge of a coarse model of other agents. We experimentally demonstrate
the efficiency of the proposed approach on a variety of case studies.