Why Online Reinforcement Learning is Causal
Oliver Schulte Pascal Poupart
Reinforcement learning (RL) and causal modelling naturally complement each
other. The goal of causal modelling is to predict the effects of interventions
in an environment, while the goal of reinforcement learning is to select
interventions that maximize the rewards the agent receives from the
environment. Reinforcement learning includes the two most powerful sources of
information for estimating causal relationships: temporal ordering and the
ability to act on an environment. This paper examines which reinforcement
learning settings we can expect to benefit from causal modelling, and how. In
online learning, the agent has the ability to interact directly with their
environment, and learn from exploring it. Our main argument is that in online
learning, conditional probabilities are causal, and therefore offline RL is the
setting where causal learning has the most potential to make a difference.
Essentially, the reason is that when an agent learns from their {\em own}
experience, there are no unobserved confounders that influence both the agent's
own exploratory actions and the rewards they receive. Our paper formalizes this
argument. For offline RL, where an agent may and typically does learn from the
experience of {\em others}, we describe previous and new methods for leveraging
a causal model, including support for counterfactual queries.