Error Discovery by Clustering Influence Embeddings
This work addresses model debugging for machine learning practitioners by providing a coherent method to discover underperforming slices, though it is incremental as it builds on existing slice discovery and influence function concepts.
The authors tackled the problem of identifying groups of test examples where a model underperforms, known as slice discovery, by introducing a method called InfEmbed that uses influence functions and clustering to ensure coherence. They demonstrated that InfEmbed outperforms state-of-the-art methods on 2 benchmarks and is effective for model debugging in case studies.
We present a method for identifying groups of test examples -- slices -- on which a model under-performs, a task now known as slice discovery. We formalize coherence -- a requirement that erroneous predictions, within a slice, should be wrong for the same reason -- as a key property that any slice discovery method should satisfy. We then use influence functions to derive a new slice discovery method, InfEmbed, which satisfies coherence by returning slices whose examples are influenced similarly by the training data. InfEmbed is simple, and consists of applying K-Means clustering to a novel representation we deem influence embeddings. We show InfEmbed outperforms current state-of-the-art methods on 2 benchmarks, and is effective for model debugging across several case studies.