Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates
This work addresses the issue of inconsistent model behavior during updates for NLP practitioners, offering methods to quantify and reduce regressions, though it is incremental in applying existing techniques like distillation and ensemble.
The paper tackles the problem of regression errors in NLP model updates, showing that regressions are prevalent across GLUE tasks and proposing a constrained optimization approach reduced to a relaxed form optimized via knowledge distillation to mitigate them.
Behavior of deep neural networks can be inconsistent between different versions. Regressions during model update are a common cause of concern that often over-weigh the benefits in accuracy or efficiency gain. This work focuses on quantifying, reducing and analyzing regression errors in the NLP model updates. Using negative flip rate as regression measure, we show that regression has a prevalent presence across tasks in the GLUE benchmark. We formulate the regression-free model updates into a constrained optimization problem, and further reduce it into a relaxed form which can be approximately optimized through knowledge distillation training method. We empirically analyze how model ensemble reduces regression. Finally, we conduct CheckList behavioral testing to understand the distribution of regressions across linguistic phenomena, and the efficacy of ensemble and distillation methods.