Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
This work addresses the need for efficient and high-performance summarization models across multiple languages, offering a significant improvement over existing large-scale models.
The paper tackles the problem of abstractive text summarization by introducing Z-Code++, a pre-trained language model that achieves new state-of-the-art results on 9 out of 13 tasks across 5 languages, outperforming much larger models like PaLM-540B and GPT3-175B in parameter efficiency.
This paper presents Z-Code++, a new pre-trained language model optimized for abstractive text summarization. The model extends the state of the art encoder-decoder model using three techniques. First, we use a two-phase pre-training process to improve model's performance on low-resource summarization tasks. The model is first pre-trained using text corpora for language understanding, and then is continually pre-trained on summarization corpora for grounded text generation. Second, we replace self-attention layers in the encoder with disentangled attention layers, where each word is represented using two vectors that encode its content and position, respectively. Third, we use fusion-in-encoder, a simple yet effective method of encoding long sequences in a hierarchical manner. Z-Code++ creates new state of the art on 9 out of 13 text summarization tasks across 5 languages. Our model is parameter-efficient in that it outperforms the 600x larger PaLM-540B on XSum, and the finetuned 200x larger GPT3-175B on SAMSum. In zero-shot and few-shot settings, our model substantially outperforms the competing models.