Measuring AI Systems Beyond Accuracy
This addresses the need for better evaluation methods in AI engineering, but it is incremental as it proposes a framework rather than a new method.
The paper tackles the problem of incomplete metrics and siloed testing in machine learning system evaluation by advocating for a robust, integrated approach, outlining six key questions to guide a holistic test and evaluation strategy.
Current test and evaluation (T&E) methods for assessing machine learning (ML) system performance often rely on incomplete metrics. Testing is additionally often siloed from the other phases of the ML system lifecycle. Research investigating cross-domain approaches to ML T&E is needed to drive the state of the art forward and to build an Artificial Intelligence (AI) engineering discipline. This paper advocates for a robust, integrated approach to testing by outlining six key questions for guiding a holistic T&E strategy.