Reasonable Scale Machine Learning with Open-Source Metaflow
This addresses productivity issues in MLOps for data practitioners across industries, though it is incremental as it builds on existing tooling concepts.
The paper tackles the challenges of developing and iterating on machine learning systems, such as reproducibility and scalability, by introducing Metaflow, an open-source framework designed to boost productivity for data practitioners.
As Machine Learning (ML) gains adoption across industries and new use cases, practitioners increasingly realize the challenges around effectively developing and iterating on ML systems: reproducibility, debugging, scalability, and documentation are elusive goals for real-world pipelines outside tech-first companies. In this paper, we review the nature of ML-oriented workloads and argue that re-purposing existing tools won't solve the current productivity issues, as ML peculiarities warrant specialized development tooling. We then introduce Metaflow, an open-source framework for ML projects explicitly designed to boost the productivity of data practitioners by abstracting away the execution of ML code from the definition of the business logic. We show how our design addresses the main challenges in ML operations (MLOps), and document through examples, interviews and use cases its practical impact on the field.