I Am Big, You Are Little; I Am Right, You Are Wrong
This provides insight into model interpretability for researchers and practitioners in computer vision, though it is incremental as it builds on existing methods for analyzing model behavior.
The paper tackled the problem of understanding how different vision models make decisions by analyzing minimal sufficient pixel sets, revealing that architectures like ConvNext and EVA have statistically different concentration patterns and that misclassified images require larger pixel sets than correct ones.
Machine learning for image classification is an active and rapidly developing field. With the proliferation of classifiers of different sizes and different architectures, the problem of choosing the right model becomes more and more important. While we can assess a model's classification accuracy statistically, our understanding of the way these models work is unfortunately limited. In order to gain insight into the decision-making process of different vision models, we propose using minimal sufficient pixels sets to gauge a model's `concentration': the pixels that capture the essence of an image through the lens of the model. By comparing position, overlap, and size of sets of pixels, we identify that different architectures have statistically different concentration, in both size and position. In particular, ConvNext and EVA models differ markedly from the others. We also identify that images which are misclassified are associated with larger pixels sets than correct classifications.