SAIBench: Benchmarking AI for Science
This work addresses the need for a unified benchmarking framework to streamline AI adoption across scientific disciplines, though it is incremental as it builds on existing benchmarking concepts.
The authors tackled the problem of scattered development and evaluation of AI solutions in scientific research by proposing SAIBench, a system that unifies scientific AI benchmarking with a domain-specific language called SAIL, enabling flexible adaptation to various problems, models, and evaluation methods.
Scientific research communities are embracing AI-based solutions to target tractable scientific tasks and improve research workflows. However, the development and evaluation of such solutions are scattered across multiple disciplines. We formalize the problem of scientific AI benchmarking, and propose a system called SAIBench in the hope of unifying the efforts and enabling low-friction on-boarding of new disciplines. The system approaches this goal with SAIL, a domain-specific language to decouple research problems, AI models, ranking criteria, and software/hardware configuration into reusable modules. We show that this approach is flexible and can adapt to problems, AI models, and evaluation methods defined in different perspectives. The project homepage is https://www.computercouncil.org/SAIBench