Asynchronous Parallel Reinforcement Learning for Optimizing Propulsive
  Performance in Fin Ray Control
Xin-Yang Liu Dariush Bodaghi Qian Xue Xudong Zheng Jian-Xun Wang
Fish fin rays constitute a sophisticated control system for ray-finned fish,
facilitating versatile locomotion within complex fluid environments. Despite
extensive research on the kinematics and hydrodynamics of fish locomotion, the
intricate control strategies in fin-ray actuation remain largely unexplored.
While deep reinforcement learning (DRL) has demonstrated potential in managing
complex nonlinear dynamics; its trial-and-error nature limits its application
to problems involving computationally demanding environmental interactions.
This study introduces a cutting-edge off-policy DRL algorithm, interacting with
a fluid-structure interaction (FSI) environment to acquire intricate fin-ray
control strategies tailored for various propulsive performance objectives. To
enhance training efficiency and enable scalable parallelism, an innovative
asynchronous parallel training (APT) strategy is proposed, which fully
decouples FSI environment interactions and policy/value network optimization.
The results demonstrated the success of the proposed method in discovering
optimal complex policies for fin-ray actuation control, resulting in a superior
propulsive performance compared to the optimal sinusoidal actuation function
identified through a parametric grid search. The merit and effectiveness of the
APT approach are also showcased through comprehensive comparison with
conventional DRL training strategies in numerical experiments of controlling
nonlinear dynamics.