A Constraint Programming-based Job Dispatcher for Modern HPC Systems and Applications
This work addresses the challenge of efficient job dispatching for HPC systems, which is incremental as it builds on existing CP-based dispatchers but improves scalability and applicability.
The paper tackles the job dispatching problem in modern HPC systems by developing a new Constraint Programming-based dispatcher that handles the entire problem in CP with a model size independent of system size, resulting in significantly increased dispatching performance in large and complex systems as shown in simulation studies.
Constraint Programming (CP) is a well-established area in AI as a programming paradigm for modelling and solving discrete optimization problems, and it has been been successfully applied to tackle the on-line job dispatching problem in HPC systems including those running modern applications. The limitations of the available CP-based job dispatchers may hinder their practical use in today's systems that are becoming larger in size and more demanding in resource allocation. In an attempt to bring basic AI research closer to a deployed application, we present a new CP-based on-line job dispatcher for modern HPC systems and applications. Unlike its predecessors, our new dispatcher tackles the entire problem in CP and its model size is independent of the system size. Experimental results based on a simulation study show that with our approach dispatching performance increases significantly in a large system and in a system where allocation is nontrivial.