Summarizing long regulatory documents with a multi-step pipeline
This work addresses the challenge of summarizing complex regulatory texts for legal professionals, but it is incremental as it builds on existing summarization methods.
The study tackled summarizing long regulatory documents by testing a multi-step extractive-abstractive pipeline, finding that its effectiveness depends on model type: it improves decoder-only models but can worsen long-context encoder-decoder models, with human evaluations favoring legal-pretrained models over general-purpose ones.
Due to their length and complexity, long regulatory texts are challenging to summarize. To address this, a multi-step extractive-abstractive architecture is proposed to handle lengthy regulatory documents more effectively. In this paper, we show that the effectiveness of a two-step architecture for summarizing long regulatory texts varies significantly depending on the model used. Specifically, the two-step architecture improves the performance of decoder-only models. For abstractive encoder-decoder models with short context lengths, the effectiveness of an extractive step varies, whereas for long-context encoder-decoder models, the extractive step worsens their performance. This research also highlights the challenges of evaluating generated texts, as evidenced by the differing results from human and automated evaluations. Most notably, human evaluations favoured language models pretrained on legal text, while automated metrics rank general-purpose language models higher. The results underscore the importance of selecting the appropriate summarization strategy based on model architecture and context length.