Human-in-the-loop: The future of Machine Learning in Automated Electron Microscopy
This work tackles the challenge of integrating real-time ML into electron microscopy operations, offering an incremental strategy to enhance automated experiments in this domain.
The paper addresses the limited use of real-time machine learning in electron microscopy by proposing human-in-the-loop automated experiments (hAE), where an ML agent controls microscope functions and a human operator guides it to achieve specific objectives.
Machine learning methods are progressively gaining acceptance in the electron microscopy community for de-noising, semantic segmentation, and dimensionality reduction of data post-acquisition. The introduction of the APIs by major instrument manufacturers now allows the deployment of ML workflows in microscopes, not only for data analytics but also for real-time decision-making and feedback for microscope operation. However, the number of use cases for real-time ML remains remarkably small. Here, we discuss some considerations in designing ML-based active experiments and pose that the likely strategy for the next several years will be human-in-the-loop automated experiments (hAE). In this paradigm, the ML learning agent directly controls beam position and image and spectroscopy acquisition functions, and human operator monitors experiment progression in real- and feature space of the system and tunes the policies of the ML agent to steer the experiment towards specific objectives.