What Is Wrong with My Model? Identifying Systematic Problems with Semantic Data Slicing
This work addresses the challenge for practitioners in diagnosing model failures by enabling more flexible and effective error analysis, though it is incremental as it builds on existing slicing methods with LLM enhancements.
The paper tackles the problem of identifying systematic errors in machine learning models by proposing SemSlicer, a framework for semantic data slicing that uses Large Language Models to generate coherent slices without relying on existing features, resulting in accurate slices with low cost and improved identification of under-performing data slices.
Machine learning models make mistakes, yet sometimes it is difficult to identify the systematic problems behind the mistakes. Practitioners engage in various activities, including error analysis, testing, auditing, and red-teaming, to form hypotheses of what can go (or has gone) wrong with their models. To validate these hypotheses, practitioners employ data slicing to identify relevant examples. However, traditional data slicing is limited by available features and programmatic slicing functions. In this work, we propose SemSlicer, a framework that supports semantic data slicing, which identifies a semantically coherent slice, without the need for existing features. SemSlicer uses Large Language Models to annotate datasets and generate slices from any user-defined slicing criteria. We show that SemSlicer generates accurate slices with low cost, allows flexible trade-offs between different design dimensions, reliably identifies under-performing data slices, and helps practitioners identify useful data slices that reflect systematic problems.