Introducing Milabench: Benchmarking Accelerators for AI
This addresses the problem of inadequate benchmarking for AI workloads in HPC systems, particularly for research communities like Mila with over 1,000 researchers, though it is incremental as it builds on existing benchmarking concepts.
The authors tackled the lack of comprehensive benchmarks for AI workloads on HPC systems by developing Milabench, a custom benchmarking suite based on a review of 867 papers and surveys, resulting in 26 primary and 16 optional benchmarks evaluated on GPUs from NVIDIA, AMD, and Intel.
AI workloads, particularly those driven by deep learning, are introducing novel usage patterns to high-performance computing (HPC) systems that are not comprehensively captured by standard HPC benchmarks. As one of the largest academic research centers dedicated to deep learning, Mila identified the need to develop a custom benchmarking suite to address the diverse requirements of its community, which consists of over 1,000 researchers. This report introduces Milabench, the resulting benchmarking suite. Its design was informed by an extensive literature review encompassing 867 papers, as well as surveys conducted with Mila researchers. This rigorous process led to the selection of 26 primary benchmarks tailored for procurement evaluations, alongside 16 optional benchmarks for in-depth analysis. We detail the design methodology, the structure of the benchmarking suite, and provide performance evaluations using GPUs from NVIDIA, AMD, and Intel. The Milabench suite is open source and can be accessed at github.com/mila-iqia/milabench.