Goodhart's Law in Reinforcement Learning
Jacek Karwowski Oliver Hayman Xingjian Bai Klaus Kiendlhofer Charlie Griffin Joar Skalse
Implementing a reward function that perfectly captures a complex task in the
real world is impractical. As a result, it is often appropriate to think of the
reward function as a proxy for the true objective rather than as its
definition. We study this phenomenon through the lens of Goodhart's law, which
predicts that increasing optimisation of an imperfect proxy beyond some
critical point decreases performance on the true objective. First, we propose a
way to quantify the magnitude of this effect and show empirically that
optimising an imperfect proxy reward often leads to the behaviour predicted by
Goodhart's law for a wide range of environments and reward functions. We then
provide a geometric explanation for why Goodhart's law occurs in Markov
decision processes. We use these theoretical insights to propose an optimal
early stopping method that provably avoids the aforementioned pitfall and
derive theoretical regret bounds for this method. Moreover, we derive a
training method that maximises worst-case reward, for the setting where there
is uncertainty about the true reward function. Finally, we evaluate our early
stopping method experimentally. Our results support a foundation for a
theoretically-principled study of reinforcement learning under reward
misspecification.