Token-Level Fitting Issues of Seq2seq Models
This addresses a fundamental training problem in seq2seq models for NLP and other deep learning tasks, though it is incremental as it builds on known early-stopping techniques.
The paper identifies that seq2seq models trained with early-stopping suffer from token-level fitting issues, where some tokens overfit while others underfit, and finds that these issues are pervasive across different models, including fine-tuned large pretrained models.
Sequence-to-sequence (seq2seq) models have been widely used for natural language processing, computer vision, and other deep learning tasks. We find that seq2seq models trained with early-stopping suffer from issues at the token level. In particular, while some tokens in the vocabulary demonstrate overfitting, others underfit when training is stopped. Experiments show that the phenomena are pervasive in different models, even in fine-tuned large pretrained-models. We identify three major factors that influence token-level fitting, which include token frequency, parts-of-speech, and prediction discrepancy. Further, we find that external factors such as language, model size, domain, data scale, and pretraining can also influence the fitting of tokens.