Theory and Practice of Quality Assurance for Machine Learning Systems An Experiment Driven Approach
This addresses the problem of project failure for practitioners developing ML systems, but it appears incremental as it builds on existing statistical control methods.
The paper tackles the challenge of ensuring quality in machine learning systems by advocating an 'experiment first' approach that uses statistical experiments to define and control key performance indicators throughout the project lifecycle, aiming to reduce failure risks.
The crafting of machine learning (ML) based systems requires statistical control throughout its life cycle. Careful quantification of business requirements and identification of key factors that impact the business requirements reduces the risk of a project failure. The quantification of business requirements results in the definition of random variables representing the system key performance indicators that need to be analyzed through statistical experiments. In addition, available data for training and experiment results impact the design of the system. Once the system is developed, it is tested and continually monitored to ensure it meets its business requirements. This is done through the continued application of statistical experiments to analyze and control the key performance indicators. This book teaches the art of crafting and developing ML based systems. It advocates an "experiment first" approach stressing the need to define statistical experiments from the beginning of the project life cycle. It also discusses in detail how to apply statistical control on the ML based system throughout its lifecycle.