Laboratory Experiments of Model-based Reinforcement Learning for
  Adaptive Optics Control
Jalo Nousiainen Byron Engler Markus Kasper Chang Rajani Tapio Helin Cédric T. Heritier Sascha P. Quanz Adrian M. Glauser
Direct imaging of Earth-like exoplanets is one of the most prominent
scientific drivers of the next generation of ground-based telescopes.
Typically, Earth-like exoplanets are located at small angular separations from
their host stars, making their detection difficult. Consequently, the adaptive
optics (AO) system's control algorithm must be carefully designed to
distinguish the exoplanet from the residual light produced by the host star.
  A new promising avenue of research to improve AO control builds on
data-driven control methods such as Reinforcement Learning (RL). RL is an
active branch of the machine learning research field, where control of a system
is learned through interaction with the environment. Thus, RL can be seen as an
automated approach to AO control, where its usage is entirely a turnkey
operation. In particular, model-based reinforcement learning (MBRL) has been
shown to cope with both temporal and misregistration errors. Similarly, it has
been demonstrated to adapt to non-linear wavefront sensing while being
efficient in training and execution.
  In this work, we implement and adapt an RL method called Policy Optimization
for AO (PO4AO) to the GHOST test bench at ESO headquarters, where we
demonstrate a strong performance of the method in a laboratory environment. Our
implementation allows the training to be performed parallel to inference, which
is crucial for on-sky operation. In particular, we study the predictive and
self-calibrating aspects of the method. The new implementation on GHOST running
PyTorch introduces only around 700 microseconds in addition to hardware,
pipeline, and Python interface latency. We open-source well-documented code for
the implementation and specify the requirements for the RTC pipeline. We also
discuss the important hyperparameters of the method, the source of the latency,
and the possible paths for a lower latency implementation.