Worrisome Properties of Neural Network Controllers and Their Symbolic
  Representations
Jacek Cyranka Kevin E M Church Jean-Philippe Lessard
We raise concerns about controllers' robustness in simple reinforcement
learning benchmark problems. We focus on neural network controllers and their
low neuron and symbolic abstractions. A typical controller reaching high mean
return values still generates an abundance of persistent low-return solutions,
which is a highly undesirable property, easily exploitable by an adversary. We
find that the simpler controllers admit more persistent bad solutions. We
provide an algorithm for a systematic robustness study and prove existence of
persistent solutions and, in some cases, periodic orbits, using a
computer-assisted proof methodology.