A Continuum of Generation Tasks for Investigating Length Bias and Degenerate Repetition
This research addresses the problem of understanding and mitigating degenerate behaviors in language models for NLP researchers, but it is incremental as it builds on prior work on task constrainedness.
The study tackled the problem of degenerate behaviors in language models, such as length bias and excessive repetition, by introducing a framework to vary task constrainedness from machine translation to open-ended generation. They found that repetition decreases with constrainedness, explaining task differences, while length bias also decreases, indicating other causes, and these issues affect only the mode of the distribution.
Language models suffer from various degenerate behaviors. These differ between tasks: machine translation (MT) exhibits length bias, while tasks like story generation exhibit excessive repetition. Recent work has attributed the difference to task constrainedness, but evidence for this claim has always involved many confounding variables. To study this question directly, we introduce a new experimental framework that allows us to smoothly vary task constrainedness, from MT at one end to fully open-ended generation at the other, while keeping all other aspects fixed. We find that: (1) repetition decreases smoothly with constrainedness, explaining the difference in repetition across tasks; (2) length bias surprisingly also decreases with constrainedness, suggesting some other cause for the difference in length bias; (3) across the board, these problems affect the mode, not the whole distribution; (4) the differences cannot be attributed to a change in the entropy of the distribution, since another method of changing the entropy, label smoothing, does not produce the same effect.