Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Design
This addresses the challenge for ML practitioners in shifting focus from implementation to problem formulation, though it is incremental as it builds on existing design practices.
The paper tackles the problem of machine learning practitioners focusing too much on low-level technical details by introducing model sketching, a framework for early-stage design that centers on high-level concepts, resulting in practitioners iterating on a broader range of designs and identifying gaps more quickly.
Machine learning practitioners often end up tunneling on low-level technical details like model architectures and performance metrics. Could early model development instead focus on high-level questions of which factors a model ought to pay attention to? Inspired by the practice of sketching in design, which distills ideas to their minimal representation, we introduce model sketching: a technical framework for iteratively and rapidly authoring functional approximations of a machine learning model's decision-making logic. Model sketching refocuses practitioner attention on composing high-level, human-understandable concepts that the model is expected to reason over (e.g., profanity, racism, or sarcasm in a content moderation task) using zero-shot concept instantiation. In an evaluation with 17 ML practitioners, model sketching reframed thinking from implementation to higher-level exploration, prompted iteration on a broader range of model designs, and helped identify gaps in the problem formulation$\unicode{x2014}$all in a fraction of the time ordinarily required to build a model.