Staged deployment of interactive multi-application HPC workflows
This work addresses the problem of simplifying HPC workflow deployment for scientific domain specialists, particularly in life sciences like neuroscience, though it appears incremental as it builds on existing solutions.
The paper tackles the complexity of deploying and executing interactive multi-application workflows on high-performance computing (HPC) systems by introducing a middleware design that extends existing workflow management solutions, enabling new scientific applications such as coupled neuronal simulators and closed-loop robot-neural network workflows.
Running scientific workflows on a supercomputer can be a daunting task for a scientific domain specialist. Workflow management solutions (WMS) are a standard method for reducing the complexity of application deployment on high performance computing (HPC) infrastructure. We introduce the design for a middleware system that extends and combines the functionality from existing solutions in order to create a high-level, staged user-centric operation/deployment model. This design addresses the requirements of several use cases in the life sciences, with a focus on neuroscience. In this manuscript we focus on two use cases: 1) three coupled neuronal simulators (for three different space/time scales) with in-transit visualization and 2) a closed-loop workflow optimized by machine learning, coupling a robot with a neural network simulation. We provide a detailed overview of the application-integrated monitoring in relationship with the HPC job. We present here a novel usage model for large scale interactive multi-application workflows running on HPC systems which aims at reducing the complexity of deployment and execution, thus enabling new science.