HIBRIDS: Attention with Hierarchical Biases for Structure-aware Long Document Summarization
This work addresses the problem of structure-aware summarization for long documents like government reports and Wikipedia articles, offering incremental improvements through a novel attention mechanism.
The authors tackled the challenge of encoding document structure into Transformer models for long document summarization by introducing HIBRIDS, which uses hierarchical biases in attention scores. Their model produced better question-summary hierarchies than comparisons, with improvements in hierarchy quality and content coverage, and also enhanced long-form summarization as measured by ROUGE scores.
Document structure is critical for efficient information consumption. However, it is challenging to encode it efficiently into the modern Transformer architecture. In this work, we present HIBRIDS, which injects Hierarchical Biases foR Incorporating Document Structure into the calculation of attention scores. We further present a new task, hierarchical question-summary generation, for summarizing salient content in the source document into a hierarchy of questions and summaries, where each follow-up question inquires about the content of its parent question-summary pair. We also annotate a new dataset with 6,153 question-summary hierarchies labeled on long government reports. Experiment results show that our model produces better question-summary hierarchies than comparisons on both hierarchy quality and content coverage, a finding also echoed by human judges. Additionally, our model improves the generation of long-form summaries from lengthy government reports and Wikipedia articles, as measured by ROUGE scores.