SPECTRUM: Speaker-Enhanced Pre-Training for Long Dialogue Summarization
This addresses the challenge of summarizing extended dialogues for applications like transcription analysis, though it appears incremental as it builds on existing pre-training methods with speaker-specific enhancements.
The paper tackled the problem of summarizing long multi-turn dialogues by proposing a speaker-enhanced pre-training method that leverages dialogue structure, and it achieved state-of-the-art performance on downstream benchmarks with long context.
Multi-turn dialogues are characterized by their extended length and the presence of turn-taking conversations. Traditional language models often overlook the distinct features of these dialogues by treating them as regular text. In this paper, we propose a speaker-enhanced pre-training method for long dialogue summarization, which leverages the inherent structure of multiple-turn dialogues. To support our study, we curate a diverse dataset that includes transcripts from real-world scenarios, movie or TV show transcripts, and dialogues generated by a Large Language Model. We then perform a pre-training, which encompasses the detection of speaker changes, and masked utterance generation. Experimental results of fine-tuned models demonstrate that our model achieves state-of-the-art performance on downstream benchmarks with long context, surpassing baseline models and highlighting the effectiveness of our approach. Our findings highlight the importance of curating pre-training datasets that exhibit diversity and variations in length distribution to ensure effective alignment with downstream datasets.