Linking the Dynamic PicoProbe Analytical Electron-Optical Beam Line / Microscope to Supercomputers
This addresses data management challenges for domain scientists at Argonne National Laboratory, but it is incremental as it builds on existing infrastructure.
The authors tackled the problem of handling high-volume data streams from the Dynamic PicoProbe microscope by developing a software architecture for large-scale data transfers to supercomputers, enabling up to 100s of GB per day and supporting workflows like machine learning and metadata extraction.
The Dynamic PicoProbe at Argonne National Laboratory is undergoing upgrades that will enable it to produce up to 100s of GB of data per day. While this data is highly important for both fundamental science and industrial applications, there is currently limited on-site infrastructure to handle these high-volume data streams. We address this problem by providing a software architecture capable of supporting large-scale data transfers to the neighboring supercomputers at the Argonne Leadership Computing Facility. To prepare for future scientific workflows, we implement two instructive use cases for hyperspectral and spatiotemporal datasets, which include: (i) off-site data transfer, (ii) machine learning/artificial intelligence and traditional data analysis approaches, and (iii) automatic metadata extraction and cataloging of experimental results. This infrastructure supports expected workloads and also provides domain scientists the ability to reinterrogate data from past experiments to yield additional scientific value and derive new insights.