Informativeness of Reward Functions in Reinforcement Learning
Rati Devidze Parameswaran Kamalaruban Adish Singla
Reward functions are central in specifying the task we want a reinforcement
learning agent to perform. Given a task and desired optimal behavior, we study
the problem of designing informative reward functions so that the designed
rewards speed up the agent's convergence. In particular, we consider
expert-driven reward design settings where an expert or teacher seeks to
provide informative and interpretable rewards to a learning agent. Existing
works have considered several different reward design formulations; however,
the key challenge is formulating a reward informativeness criterion that adapts
w.r.t. the agent's current policy and can be optimized under specified
structural constraints to obtain interpretable rewards. In this paper, we
propose a novel reward informativeness criterion, a quantitative measure that
captures how the agent's current policy will improve if it receives rewards
from a specific reward function. We theoretically showcase the utility of the
proposed informativeness criterion for adaptively designing rewards for an
agent. Experimental results on two navigation tasks demonstrate the
effectiveness of our adaptive reward informativeness criterion.