ModelDiff: A Framework for Comparing Learning Algorithms
This work addresses the need for systematic comparison of learning algorithms for researchers and practitioners, but it is incremental as it builds on existing datamodels frameworks.
The paper tackles the problem of comparing learning algorithms by formalizing it as finding distinguishing feature transformations and presents ModelDiff, a method that uses the datamodels framework to analyze how algorithms use training data, demonstrated through case studies on data augmentation, pre-training, and SGD hyperparameters.
We study the problem of (learning) algorithm comparison, where the goal is to find differences between models trained with two different learning algorithms. We begin by formalizing this goal as one of finding distinguishing feature transformations, i.e., input transformations that change the predictions of models trained with one learning algorithm but not the other. We then present ModelDiff, a method that leverages the datamodels framework (Ilyas et al., 2022) to compare learning algorithms based on how they use their training data. We demonstrate ModelDiff through three case studies, comparing models trained with/without data augmentation, with/without pre-training, and with different SGD hyperparameters. Our code is available at https://github.com/MadryLab/modeldiff .