Reliable and Interpretable Drift Detection in Streams of Short Texts
This addresses the issue of maintaining model performance over time in dialog systems for customers, though it appears incremental as it builds on existing drift detection concepts.
The authors tackled the problem of detecting and interpreting data drift in short text streams, which causes model performance degradation, by proposing an end-to-end framework for reliable model-agnostic change-point detection and interpretation in large task-oriented dialog systems, proven effective in multiple customer deployments.
Data drift is the change in model input data that is one of the key factors leading to machine learning models performance degradation over time. Monitoring drift helps detecting these issues and preventing their harmful consequences. Meaningful drift interpretation is a fundamental step towards effective re-training of the model. In this study we propose an end-to-end framework for reliable model-agnostic change-point detection and interpretation in large task-oriented dialog systems, proven effective in multiple customer deployments. We evaluate our approach and demonstrate its benefits with a novel variant of intent classification training dataset, simulating customer requests to a dialog system. We make the data publicly available.