Solving Deep Reinforcement Learning Benchmarks with Linear Policy
  Networks
Annie Wong Jacob de Nobel Thomas Bäck Aske Plaat Anna V. Kononova
Although Deep Reinforcement Learning (DRL) methods can learn effective
policies for challenging problems such as Atari games and robotics tasks,
algorithms are complex and training times are often long. This study
investigates how evolution strategies (ES) perform compared to gradient-based
deep reinforcement learning methods. We use ES to optimize the weights of a
neural network via neuroevolution, performing direct policy search. We
benchmark both regular networks and policy networks consisting of a single
linear layer from observations to actions; for three classical ES methods and
for three gradient-based methods such as PPO. Our results reveal that ES can
find effective linear policies for many RL benchmark tasks, in contrast to DRL
methods that can only find successful policies using much larger networks,
suggesting that current benchmarks are easier to solve than previously assumed.
Interestingly, also for higher complexity tasks, ES achieves results comparable
to gradient-based DRL algorithms. Furthermore, we find that by directly
accessing the memory state of the game, ES are able to find successful policies
in Atari, outperforming DQN. While gradient-based methods have dominated the
field in recent years, ES offers an alternative that is easy to implement,
parallelize, understand, and tune.