Contrast transfer functions help quantify neural network out-of-distribution generalization in HRTEM
This addresses the challenge of deploying neural networks in experimental workflows where ground truth is hard to establish, but it is incremental as it focuses on a specific domain and imaging condition shifts.
The study tackled the problem of neural network out-of-distribution (OOD) generalization in high-resolution transmission electron microscopy (HRTEM) segmentation by training over 12,000 models on synthetic data and found that performance smoothly and predictably degrades as imaging conditions shift from the training distribution.
Neural networks, while effective for tackling many challenging scientific tasks, are not known to perform well out-of-distribution (OOD), i.e., within domains which differ from their training data. Understanding neural network OOD generalization is paramount to their successful deployment in experimental workflows, especially when ground-truth knowledge about the experiment is hard to establish or experimental conditions significantly vary. With inherent access to ground-truth information and fine-grained control of underlying distributions, simulation-based data curation facilitates precise investigation of OOD generalization behavior. Here, we probe generalization with respect to imaging conditions of neural network segmentation models for high-resolution transmission electron microscopy (HRTEM) imaging of nanoparticles, training and measuring the OOD generalization of over 12,000 neural networks using synthetic data generated via random structure sampling and multislice simulation. Using the HRTEM contrast transfer function, we further develop a framework to compare information content of HRTEM datasets and quantify OOD domain shifts. We demonstrate that neural network segmentation models enjoy significant performance stability, but will smoothly and predictably worsen as imaging conditions shift from the training distribution. Lastly, we consider limitations of our approach in explaining other OOD shifts, such as of the atomic structures, and discuss complementary techniques for understanding generalization in such settings.