Scalable Runtime Architecture for Data-driven, Hybrid HPC and ML Workflow Applications
This work addresses the need for scalable and efficient hybrid HPC/ML workflows in scientific computing, though it appears incremental as it builds on existing RADICAL-Pilot technology.
The paper tackles the challenge of integrating traditional HPC with ML workflows by developing a scalable runtime system based on RADICAL-Pilot, which enables distributed ML execution and efficient resource management across platforms, with preliminary results showing minimal overhead in concurrent model execution.
Hybrid workflows combining traditional HPC and novel ML methodologies are transforming scientific computing. This paper presents the architecture and implementation of a scalable runtime system that extends RADICAL-Pilot with service-based execution to support AI-out-HPC workflows. Our runtime system enables distributed ML capabilities, efficient resource management, and seamless HPC/ML coupling across local and remote platforms. Preliminary experimental results show that our approach manages concurrent execution of ML models across local and remote HPC/cloud resources with minimal architectural overheads. This lays the foundation for prototyping three representative data-driven workflow applications and executing them at scale on leadership-class HPC platforms.