Synthesizing Programmatic Policies with Actor-Critic Algorithms and ReLU
  Networks
Spyros Orfanos Levi H. S. Lelis
Programmatically Interpretable Reinforcement Learning (PIRL) encodes policies
in human-readable computer programs. Novel algorithms were recently introduced
with the goal of handling the lack of gradient signal to guide the search in
the space of programmatic policies. Most of such PIRL algorithms first train a
neural policy that is used as an oracle to guide the search in the programmatic
space. In this paper, we show that such PIRL-specific algorithms are not
needed, depending on the language used to encode the programmatic policies.
This is because one can use actor-critic algorithms to directly obtain a
programmatic policy. We use a connection between ReLU neural networks and
oblique decision trees to translate the policy learned with actor-critic
algorithms into programmatic policies. This translation from ReLU networks
allows us to synthesize policies encoded in programs with if-then-else
structures, linear transformations of the input values, and PID operations.
Empirical results on several control problems show that this translation
approach is capable of learning short and effective policies. Moreover, the
translated policies are at least competitive and often far superior to the
policies PIRL algorithms synthesize.