Benchmarking Contemporary Deep Learning Hardware and Frameworks:A Survey of Qualitative Metrics
It provides a qualitative comparison for researchers and practitioners in deep learning hardware and frameworks, but is incremental as it synthesizes existing information without new experimental results.
This paper surveys benchmarking principles, hardware devices (GPUs, FPGAs, ASICs), and deep learning software frameworks, using a 6-metric approach for frameworks and an 11-metric approach for hardware, and mentions MLPerf benchmark results and metrics.
This paper surveys benchmarking principles, machine learning devices including GPUs, FPGAs, and ASICs, and deep learning software frameworks. It also reviews these technologies with respect to benchmarking from the perspectives of a 6-metric approach to frameworks and an 11-metric approach to hardware platforms. Because MLPerf is a benchmark organization working with industry and academia, and offering deep learning benchmarks that evaluate training and inference on deep learning hardware devices, the survey also mentions MLPerf benchmark results, benchmark metrics, datasets, deep learning frameworks and algorithms. We summarize seven benchmarking principles, differential characteristics of mainstream AI devices, and qualitative comparison of deep learning hardware and frameworks.