Protected probabilistic classification
This work addresses the challenge of maintaining model quality in real-world machine learning applications where data distributions shift over time, though it appears incremental by building on existing methods.
The paper tackles the problem of protecting probabilistic classification models from performance degradation due to changes in data distribution, focusing on binary classification. It proposes techniques based on conformal test martingales and prediction with expert advice to address this issue.
This paper proposes a way of protecting probabilistic prediction models against changes in the data distribution, concentrating on the case of classification and paying particular attention to binary classification. This is important in applications of machine learning, where the quality of a trained prediction algorithm may drop significantly in the process of its exploitation. Our techniques are based on recent work on conformal test martingales and older work on prediction with expert advice, namely tracking the best expert.