Continuous benchmarking: Keeping pace with an evolving ecosystem of models and technologies
For researchers in neuroscience and AI, this provides a software-engineering approach to systematic benchmarking, but it is an incremental extension of prior work.
The authors propose an automated benchmarking pipeline inspired by continuous integration to keep pace with evolving models and HPC systems, targeting reproducibility and re-use in neuroscience and AI.
Drawing on ideas from continuous integration, we present concepts of an automated benchmarking pipeline for high performance applications. Customization and collaboration have been key design goals owing to the requirements of research-software development as a continuous community effort. We have extended our previous conceptual work on systematic benchmarking workflows with the functionality of user-agnostic operations as well as continuous benchmarking. This fosters reproducibility and re-use of benchmarking results to ensure sustainable technological progress. We provide software-engineering solutions to keep pace with the rapid evolution of both large-scale models and high-performance computing systems with a view towards the scientific domains of neuroscience and artificial intelligence.