Retrain or not retrain: Conformal test martingales for change-point detection
This addresses the challenge of maintaining model performance over time for users of prediction algorithms, though it appears incremental as it builds on existing conformal prediction methods.
The paper tackles the problem of detecting when a prediction algorithm needs retraining due to data distribution changes, using conformal test martingales based on exchangeability, with validity guaranteed and initial efficiency exploration.
We argue for supplementing the process of training a prediction algorithm by setting up a scheme for detecting the moment when the distribution of the data changes and the algorithm needs to be retrained. Our proposed schemes are based on exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the data. Our method, based on conformal prediction, is general and can be applied on top of any modern prediction algorithm. Its validity is guaranteed, and in this paper we make first steps in exploring its efficiency.