MLTEing Models: Negotiating, Evaluating, and Documenting Model and System Qualities
This addresses the problem for organizations and interdisciplinary teams needing to evaluate ML models and systems in production, but it is incremental as it builds on existing evaluation techniques.
The paper tackles the lack of a cohesive methodology for ensuring ML/AI systems work as intended in production by proposing MLTE, a framework and implementation that compiles state-of-the-art evaluation techniques into an organizational process, resulting in tooling that supports requirement expression, metric collection, and result communication.
Many organizations seek to ensure that machine learning (ML) and artificial intelligence (AI) systems work as intended in production but currently do not have a cohesive methodology in place to do so. To fill this gap, we propose MLTE (Machine Learning Test and Evaluation, colloquially referred to as "melt"), a framework and implementation to evaluate ML models and systems. The framework compiles state-of-the-art evaluation techniques into an organizational process for interdisciplinary teams, including model developers, software engineers, system owners, and other stakeholders. MLTE tooling supports this process by providing a domain-specific language that teams can use to express model requirements, an infrastructure to define, generate, and collect ML evaluation metrics, and the means to communicate results.