GraphNet: A Large-Scale Computational Graph Dataset for Tensor Compiler Research
This provides a dataset and metrics for tensor compiler research, addressing the need for standardized evaluation in optimizing deep learning computations, though it is incremental as it builds on existing compiler tools.
The authors introduced GraphNet, a dataset of 2.7K real-world deep learning computational graphs with metadata across six task categories, and proposed benchmark metrics like Speedup Score S(t) and Error-aware Speedup Score ES(t) to evaluate tensor compiler performance, demonstrating practicality by benchmarking default compilers on CV and NLP samples.
We introduce GraphNet, a dataset of 2.7K real-world deep learning computational graphs with rich metadata, spanning six major task categories across multiple deep learning frameworks. To evaluate tensor compiler performance on these samples, we propose the benchmark metric Speedup Score S(t), which jointly considers runtime speedup and execution correctness under tunable tolerance levels, offering a reliable measure of general optimization capability. Furthermore, we extend S(t) to the Error-aware Speedup Score ES(t), which incorporates error information and helps compiler developers identify key performance bottlenecks. In this report, we benchmark the default tensor compilers, CINN for PaddlePaddle and TorchInductor for PyTorch, on computer vision (CV) and natural language processing (NLP) samples to demonstrate the practicality of GraphNet. The full construction pipeline with graph extraction and compiler evaluation tools is available at https://github.com/PaddlePaddle/GraphNet .