Distributionally Robust Group Backwards Compatibility
This work addresses fairness concerns for underrepresented groups in machine learning model updates, though it is incremental as it builds on existing robustness and fairness techniques.
The paper tackles the problem of backward compatibility errors in updated machine learning models, particularly affecting underrepresented groups, by proposing methods based on distributional robustness and minimax fairness. Experimental results on CIFAR-10, CelebA, and Waterbirds datasets demonstrate the effectiveness of these approaches in mitigating performance disparities.
Machine learning models are updated as new data is acquired or new architectures are developed. These updates usually increase model performance, but may introduce backward compatibility errors, where individual users or groups of users see their performance on the updated model adversely affected. This problem can also be present when training datasets do not accurately reflect overall population demographics, with some groups having overall lower participation in the data collection process, posing a significant fairness concern. We analyze how ideas from distributional robustness and minimax fairness can aid backward compatibility in this scenario, and propose two methods to directly address this issue. Our theoretical analysis is backed by experimental results on CIFAR-10, CelebA, and Waterbirds, three standard image classification datasets. Code available at github.com/natalialmg/GroupBC