VirnyFlow: A Design Space for Responsible Model Development
This addresses the need for more flexible and responsible automation in ML development for data scientists, though it appears incremental as an extension of AutoML with customization features.
The paper tackles the problem of developing machine learning models that are tailored to real-world, multi-objective contexts by introducing VirnyFlow, a design space for responsible model development. It shows that VirnyFlow significantly outperforms state-of-the-art AutoML systems in optimization quality and scalability across five real-world benchmarks.
Developing machine learning (ML) models requires a deep understanding of real-world problems, which are inherently multi-objective. In this paper, we present VirnyFlow, the first design space for responsible model development, designed to assist data scientists in building ML pipelines that are tailored to the specific context of their problem. Unlike conventional AutoML frameworks, VirnyFlow enables users to define customized optimization criteria, perform comprehensive experimentation across pipeline stages, and iteratively refine models in alignment with real-world constraints. Our system integrates evaluation protocol definition, multi-objective Bayesian optimization, cost-aware multi-armed bandits, query optimization, and distributed parallelism into a unified architecture. We show that VirnyFlow significantly outperforms state-of-the-art AutoML systems in both optimization quality and scalability across five real-world benchmarks, offering a flexible, efficient, and responsible alternative to black-box automation in ML development.