Program-Based Strategy Induction for Reinforcement Learning
Carlos G. Correa Thomas L. Griffiths Nathaniel D. Daw
Typical models of learning assume incremental estimation of
continuously-varying decision variables like expected rewards. However, this
class of models fails to capture more idiosyncratic, discrete heuristics and
strategies that people and animals appear to exhibit. Despite recent advances
in strategy discovery using tools like recurrent networks that generalize the
classic models, the resulting strategies are often onerous to interpret, making
connections to cognition difficult to establish. We use Bayesian program
induction to discover strategies implemented by programs, letting the
simplicity of strategies trade off against their effectiveness. Focusing on
bandit tasks, we find strategies that are difficult or unexpected with
classical incremental learning, like asymmetric learning from rewarded and
unrewarded trials, adaptive horizon-dependent random exploration, and discrete
state switching.