Human-in-the-loop: The future of Machine Learning in Automated Electron
  Microscopy
Sergei V. Kalinin Yongtao Liu Arpan Biswas Gerd Duscher Utkarsh Pratiush Kevin Roccapriore Maxim Ziatdinov Rama Vasudevan
Machine learning methods are progressively gaining acceptance in the electron
microscopy community for de-noising, semantic segmentation, and dimensionality
reduction of data post-acquisition. The introduction of the APIs by major
instrument manufacturers now allows the deployment of ML workflows in
microscopes, not only for data analytics but also for real-time decision-making
and feedback for microscope operation. However, the number of use cases for
real-time ML remains remarkably small. Here, we discuss some considerations in
designing ML-based active experiments and pose that the likely strategy for the
next several years will be human-in-the-loop automated experiments (hAE). In
this paradigm, the ML learning agent directly controls beam position and image
and spectroscopy acquisition functions, and human operator monitors experiment
progression in real- and feature space of the system and tunes the policies of
the ML agent to steer the experiment towards specific objectives.