Few-Shot Learning of an Interleaved Text Summarization Model by Pretraining with Synthetic Data
This addresses the challenge of quickly obtaining overviews from interleaved discussions for users, though it is incremental as it builds on existing end-to-end and pretraining methods.
The paper tackled the problem of summarizing interleaved texts, such as online chats, by proposing an end-to-end hierarchical encoder-decoder system pretrained with synthetic data, which outperformed a traditional two-step system by 22% on a real-world meeting dataset.
Interleaved texts, where posts belonging to different threads occur in a sequence, commonly occur in online chat posts, so that it can be time-consuming to quickly obtain an overview of the discussions. Existing systems first disentangle the posts by threads and then extract summaries from those threads. A major issue with such systems is error propagation from the disentanglement component. While end-to-end trainable summarization system could obviate explicit disentanglement, such systems require a large amount of labeled data. To address this, we propose to pretrain an end-to-end trainable hierarchical encoder-decoder system using synthetic interleaved texts. We show that by fine-tuning on a real-world meeting dataset (AMI), such a system out-performs a traditional two-step system by 22%. We also compare against transformer models and observed that pretraining with synthetic data both the encoder and decoder outperforms the BertSumExtAbs transformer model which pretrains only the encoder on a large dataset.