Measuring and Reducing Model Update Regression in Structured Prediction for NLP
This addresses backward compatibility for industrial NLP applications, focusing on structured prediction tasks like dependency parsing, but it is incremental as it builds on existing techniques.
The paper tackles the problem of model update regression in structured prediction for NLP, where new models may perform worse on cases previously handled correctly, and proposes Backward-Congruent Re-ranking (BCR) to reduce this regression, showing it outperforms existing methods like model ensemble and knowledge distillation.
Recent advance in deep learning has led to the rapid adoption of machine learning-based NLP models in a wide range of applications. Despite the continuous gain in accuracy, backward compatibility is also an important aspect for industrial applications, yet it received little research attention. Backward compatibility requires that the new model does not regress on cases that were correctly handled by its predecessor. This work studies model update regression in structured prediction tasks. We choose syntactic dependency parsing and conversational semantic parsing as representative examples of structured prediction tasks in NLP. First, we measure and analyze model update regression in different model update settings. Next, we explore and benchmark existing techniques for reducing model update regression including model ensemble and knowledge distillation. We further propose a simple and effective method, Backward-Congruent Re-ranking (BCR), by taking into account the characteristics of structured prediction. Experiments show that BCR can better mitigate model update regression than model ensemble and knowledge distillation approaches.