UniPCM: Universal Pre-trained Conversation Model with Task-aware Automatic Prompt
This work addresses the challenge of prompt quality and scalability in building robust dialog systems, representing a novel method for a known bottleneck rather than a foundational advancement.
The authors tackled the problem of suboptimal human-defined prompts in multi-task pre-training for dialog systems by proposing Task-based Automatic Prompt generation (TAP) to automatically generate high-quality prompts, resulting in UniPCM, which achieved state-of-the-art results on 9 datasets and strong performance in low-resource scenarios.
Recent research has shown that multi-task pre-training greatly improves the model's robustness and transfer ability, which is crucial for building a high-quality dialog system. However, most previous works on multi-task pre-training rely heavily on human-defined input format or prompt, which is not optimal in quality and quantity. In this work, we propose to use Task-based Automatic Prompt generation (TAP) to automatically generate high-quality prompts. Using the high-quality prompts generated, we scale the corpus of the pre-trained conversation model to 122 datasets from 15 dialog-related tasks, resulting in Universal Pre-trained Conversation Model (UniPCM), a powerful foundation model for various conversational tasks and different dialog systems. Extensive experiments have shown that UniPCM is robust to input prompts and capable of various dialog-related tasks. Moreover, UniPCM has strong transfer ability and excels at low resource scenarios, achieving SOTA results on 9 different datasets ranging from task-oriented dialog to open-domain conversation. Furthermore, we are amazed to find that TAP can generate prompts on par with those collected with crowdsourcing. The code is released with the paper.