Duration-Informed Workload Scheduler
This work addresses scheduling efficiency for users and systems in supercomputing, representing an incremental improvement.
The paper tackled the problem of job scheduling in high-performance computing by enhancing a workload scheduler with a machine learning-based duration prediction module, resulting in an 11% decrease in mean waiting time for jobs.
High-performance computing systems are complex machines whose behaviour is governed by the correct functioning of its many subsystems. Among these, the workload scheduler has a crucial impact on the timely execution of the jobs continuously submitted to the computing resources. Making high-quality scheduling decisions is contingent on knowing the duration of submitted jobs before their execution--a non-trivial task for users that can be tackled with Machine Learning. In this work, we devise a workload scheduler enhanced with a duration prediction module built via Machine Learning. We evaluate its effectiveness and show its performance using workload traces from a Tier-0 supercomputer, demonstrating a decrease in mean waiting time across all jobs of around 11%. Lower waiting times are directly connected to better quality of service from the users' point of view and higher turnaround from the system's perspective.