AIBench Scenario: Scenario-distilling AI Benchmarking
This addresses benchmarking challenges for AI developers and researchers in complex real-world systems, though it is incremental as it builds on existing benchmarking concepts.
The paper tackles the challenge of benchmarking AI in complex real-world applications by proposing a methodology to distill scenarios into essential AI and non-AI tasks, creating scenario benchmarks; it demonstrates the advantage of this approach over using component or micro benchmarks alone, with nine benchmarks extracted from industry partners.
Modern real-world application scenarios like Internet services consist of a diversity of AI and non-AI modules with huge code sizes and long and complicated execution paths, which raises serious benchmarking or evaluating challenges. Using AI components or micro benchmarks alone can lead to error-prone conclusions. This paper presents a methodology to attack the above challenge. We formalize a real-world application scenario as a Directed Acyclic Graph-based model and propose the rules to distill it into a permutation of essential AI and non-AI tasks, which we call a scenario benchmark. Together with seventeen industry partners, we extract nine typical scenario benchmarks. We design and implement an extensible, configurable, and flexible benchmark framework. We implement two Internet service AI scenario benchmarks based on the framework as proxies to two real-world application scenarios. We consider scenario, component, and micro benchmarks as three indispensable parts for evaluating. Our evaluation shows the advantage of our methodology against using component or micro AI benchmarks alone. The specifications, source code, testbed, and results are publicly available from \url{https://www.benchcouncil.org/aibench/scenario/}.