How Much Consistency Is Your Accuracy Worth?
This work addresses the need for better robustness metrics in machine learning, offering a tool to refine consistency evaluations, though it is incremental as it builds on existing contrast set methods.
The authors tackled the problem of evaluating model robustness through contrast set consistency, proposing relative consistency as a complementary measure to assess if a model's consistency is optimal for its accuracy, and found that it can alter comparative assessments of models.
Contrast set consistency is a robustness measurement that evaluates the rate at which a model correctly responds to all instances in a bundle of minimally different examples relying on the same knowledge. To draw additional insights, we propose to complement consistency with relative consistency -- the probability that an equally accurate model would surpass the consistency of the proposed model, given a distribution over possible consistencies. Models with 100% relative consistency have reached a consistency peak for their accuracy. We reflect on prior work that reports consistency in contrast sets and observe that relative consistency can alter the assessment of a model's consistency compared to another. We anticipate that our proposed measurement and insights will influence future studies aiming to promote consistent behavior in models.