Model Stability with Continuous Data Updates
This addresses stability issues for NLP practitioners using continuous training, but it is incremental as it builds on existing methods without major breakthroughs.
The paper tackles the problem of model stability in NLP systems with continuous data updates, finding that non-RNN models and fastText embeddings improve stability, while ensemble and incremental training strategies also significantly influence it.
In this paper, we study the "stability" of machine learning (ML) models within the context of larger, complex NLP systems with continuous training data updates. For this study, we propose a methodology for the assessment of model stability (which we refer to as jitter under various experimental conditions. We find that model design choices, including network architecture and input representation, have a critical impact on stability through experiments on four text classification tasks and two sequence labeling tasks. In classification tasks, non-RNN-based models are observed to be more stable than RNN-based ones, while the encoder-decoder model is less stable in sequence labeling tasks. Moreover, input representations based on pre-trained fastText embeddings contribute to more stability than other choices. We also show that two learning strategies -- ensemble models and incremental training -- have a significant influence on stability. We recommend ML model designers account for trade-offs in accuracy and jitter when making modeling choices.