An Empirical Study on Context Length for Open-Domain Dialog Generation
This work addresses a practical issue for developers of dialog systems, but it is incremental as it focuses on optimizing an existing model parameter without introducing new methods.
The study investigated how context length affects Transformer-based open-domain dialog models, finding that this often overlooked setting significantly impacts model performance and deserves careful consideration.
Transformer-based open-domain dialog models have become increasingly popular in recent years. These models typically represent context as a concatenation of a dialog history. However, there is no criterion to decide how many utterances should be kept adequate in a context. We try to figure out how the choice of context length affects the model. We experiment on three questions from coarse to fine: (i) Does longer context help model training? (ii) Is it necessary to change the training context length when dealing with dialogs of different context lengths? (iii) Do different dialog samples have the same preference for context length? Our experimental results show that context length, an often overlooked setting, deserves attention when implementing Transformer-based dialog models.