Probing Task-Oriented Dialogue Representation from Language Models
Provides guidelines for selecting pre-trained language models for dialogue research, which is useful for researchers in natural language processing and dialogue systems.
This paper investigates which pre-trained language models provide the most informative representations for task-oriented dialogue tasks, using both supervised classifier probes and a novel unsupervised mutual information probe to evaluate model performance.
This paper investigates pre-trained language models to find out which model intrinsically carries the most informative representation for task-oriented dialogue tasks. We approach the problem from two aspects: supervised classifier probe and unsupervised mutual information probe. We fine-tune a feed-forward layer as the classifier probe on top of a fixed pre-trained language model with annotated labels in a supervised way. Meanwhile, we propose an unsupervised mutual information probe to evaluate the mutual dependence between a real clustering and a representation clustering. The goals of this empirical paper are to 1) investigate probing techniques, especially from the unsupervised mutual information aspect, 2) provide guidelines of pre-trained language model selection for the dialogue research community, 3) find insights of pre-training factors for dialogue application that may be the key to success.