Testing for concept shift online
This work addresses the problem of detecting concept shift in data streams for machine learning practitioners, which is an incremental improvement to existing shift detection methods.
This paper introduces exchangeability martingales for online detection of concept shift, a violation of the IID assumption common in machine learning. The authors propose methods based on conformal prediction and also discuss martingales that decompose into concept shift and label shift detection components.
This note continues study of exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the observations. Such processes can be used for detecting violations of the IID assumption, which is commonly made in machine learning. Violations of the IID assumption are sometimes referred to as dataset shift, and dataset shift is sometimes subdivided into concept shift, covariate shift, etc. Our primary interest is in concept shift, but we will also discuss exchangeability martingales that decompose perfectly into two components one of which detects concept shift and the other detects what we call label shift. Our methods will be based on techniques of conformal prediction.