Detecting model drift using polynomial relations
This addresses the issue of model performance degradation due to distribution shifts for practitioners relying on stable ML predictions, though it appears incremental as it builds on existing drift detection methods with a new polynomial-based technique.
The paper tackles the problem of detecting data distribution drift in machine learning models by identifying strong polynomial relations between features and measuring their stability using Bayes Factors, empirically validating the approach through simulations of various changes.
Machine learning models serve critical functions, such as classifying loan applicants as good or bad risks. Each model is trained under the assumption that the data used in training and in the field come from the same underlying unknown distribution. Often, this assumption is broken in practice. It is desirable to identify when this occurs, to minimize the impact on model performance. We suggest a new approach to detecting change in the data distribution by identifying polynomial relations between the data features. We measure the strength of each identified relation using its R-square value. A strong polynomial relation captures a significant trait of the data which should remain stable if the data distribution does not change. We thus use a set of learned strong polynomial relations to identify drift. For a set of polynomial relations that are stronger than a given threshold, we calculate the amount of drift observed for that relation. The amount of drift is measured by calculating the Bayes Factor for the polynomial relation likelihood of the baseline data versus field data. We empirically validate the approach by simulating a range of changes, and identify drift using the Bayes Factor of the polynomial relation likelihood change.