Accelerating scientific discovery with the common task framework
This addresses the need for standardized evaluation in science and engineering, but it is incremental as it adapts an existing framework from traditional ML domains.
The paper tackles the problem of evaluating diverse machine learning algorithms in scientific and engineering applications by introducing a common task framework (CTF) with challenge datasets and objective metrics, aiming to accelerate discovery without specifying concrete numerical results.
Machine learning (ML) and artificial intelligence (AI) algorithms are transforming and empowering the characterization and control of dynamic systems in the engineering, physical, and biological sciences. These emerging modeling paradigms require comparative metrics to evaluate a diverse set of scientific objectives, including forecasting, state reconstruction, generalization, and control, while also considering limited data scenarios and noisy measurements. We introduce a common task framework (CTF) for science and engineering, which features a growing collection of challenge data sets with a diverse set of practical and common objectives. The CTF is a critically enabling technology that has contributed to the rapid advance of ML/AI algorithms in traditional applications such as speech recognition, language processing, and computer vision. There is a critical need for the objective metrics of a CTF to compare the diverse algorithms being rapidly developed and deployed in practice today across science and engineering.