"No, they did not": Dialogue response dynamics in pre-trained language models
This work addresses the problem of dialogue competence in language models for NLP applications, but it is incremental as it reveals limitations rather than proposing solutions.
The paper investigated whether pre-trained language models can appropriately respond to dialogue dynamics involving at-issueness and ellipsis, finding that models show sensitivity to embedded clauses but have mixed and weak performance in capturing these dynamics, with fundamental limitations in ellipsis understanding.
A critical component of competence in language is being able to identify relevant components of an utterance and reply appropriately. In this paper we examine the extent of such dialogue response sensitivity in pre-trained language models, conducting a series of experiments with a particular focus on sensitivity to dynamics involving phenomena of at-issueness and ellipsis. We find that models show clear sensitivity to a distinctive role of embedded clauses, and a general preference for responses that target main clause content of prior utterances. However, the results indicate mixed and generally weak trends with respect to capturing the full range of dynamics involved in targeting at-issue versus not-at-issue content. Additionally, models show fundamental limitations in grasp of the dynamics governing ellipsis, and response selections show clear interference from superficial factors that outweigh the influence of principled discourse constraints.