Function Approximation for Reinforcement Learning Controller for Energy
  from Spread Waves
Soumyendu Sarkar Vineet Gundecha Sahand Ghorbanpour Alexander Shmakov Ashwin Ramesh Babu Avisek Naug Alexandre Pichard Mathieu Cocho
The industrial multi-generator Wave Energy Converters (WEC) must handle
multiple simultaneous waves coming from different directions called spread
waves. These complex devices in challenging circumstances need controllers with
multiple objectives of energy capture efficiency, reduction of structural
stress to limit maintenance, and proactive protection against high waves. The
Multi-Agent Reinforcement Learning (MARL) controller trained with the Proximal
Policy Optimization (PPO) algorithm can handle these complexities. In this
paper, we explore different function approximations for the policy and critic
networks in modeling the sequential nature of the system dynamics and find that
they are key to better performance. We investigated the performance of a fully
connected neural network (FCN), LSTM, and Transformer model variants with
varying depths and gated residual connections. Our results show that the
transformer model of moderate depth with gated residual connections around the
multi-head attention, multi-layer perceptron, and the transformer block (STrXL)
proposed in this paper is optimal and boosts energy efficiency by an average of
22.1% for these complex spread waves over the existing spring damper (SD)
controller. Furthermore, unlike the default SD controller, the transformer
controller almost eliminated the mechanical stress from the rotational yaw
motion for angled waves. Demo: https://tinyurl.com/yueda3jh