HPC AI500: A Benchmark Suite for HPC AI Systems
This work addresses the need for standardized evaluation metrics for HPC AI systems in scientific computing, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the challenge of evaluating high-performance computing (HPC) systems for scientific deep learning workloads by proposing HPC AI500, a benchmark suite with 14 real-world scientific DL applications, and they provided a scalable reference implementation as part of an open-source project.
In recent years, with the trend of applying deep learning (DL) in high performance scientific computing, the unique characteristics of emerging DL workloads in HPC raise great challenges in designing, implementing HPC AI systems. The community needs a new yard stick for evaluating the future HPC systems. In this paper, we propose HPC AI500 --- a benchmark suite for evaluating HPC systems that running scientific DL workloads. Covering the most representative scientific fields, each workload from HPC AI500 is based on real-world scientific DL applications. Currently, we choose 14 scientific DL benchmarks from perspectives of application scenarios, data sets, and software stack. We propose a set of metrics for comprehensively evaluating the HPC AI systems, considering both accuracy, performance as well as power and cost. We provide a scalable reference implementation of HPC AI500. HPC AI500 is a part of the open-source AIBench project, the specification and source code are publicly available from \url{http://www.benchcouncil.org/AIBench/index.html}.