ChuLo: Chunk-Level Key Information Representation for Long Document Understanding
This addresses the challenge of handling long documents in NLP for tasks requiring fine-grained annotations, though it is an incremental improvement over existing chunking methods.
The paper tackles the problem of computational limitations in Transformer-based models for long document understanding by introducing ChuLo, a chunk representation method that uses unsupervised keyphrase extraction to group tokens, reducing input length while minimizing information loss, and demonstrates effectiveness on multiple long document classification and token classification tasks.
Transformer-based models have achieved remarkable success in various Natural Language Processing (NLP) tasks, yet their ability to handle long documents is constrained by computational limitations. Traditional approaches, such as truncating inputs, sparse self-attention, and chunking, attempt to mitigate these issues, but they often lead to information loss and hinder the model's ability to capture long-range dependencies. In this paper, we introduce ChuLo, a novel chunk representation method for long document understanding that addresses these limitations. Our ChuLo groups input tokens using unsupervised keyphrase extraction, emphasizing semantically important keyphrase based chunks to retain core document content while reducing input length. This approach minimizes information loss and improves the efficiency of Transformer-based models. Preserving all tokens in long document understanding, especially token classification tasks, is important to ensure that fine-grained annotations, which depend on the entire sequence context, are not lost. We evaluate our method on multiple long document classification tasks and long document token classification tasks, demonstrating its effectiveness through comprehensive qualitative and quantitative analysis. Our implementation is open-sourced on https://github.com/adlnlp/Chulo.