Long-Range Transformer Architectures for Document Understanding
This work addresses the computational limitations of Transformers for long sequences in Document Understanding, offering incremental improvements for processing multi-page documents.
The paper tackles the problem of applying Transformer models to long multi-page documents in Document Understanding by introducing two new multi-modal long-range models and a 2D relative attention bias, achieving improvements on multi-page business documents for Information Retrieval with a small performance cost on smaller sequences.
Since their release, Transformers have revolutionized many fields from Natural Language Understanding to Computer Vision. Document Understanding (DU) was not left behind with first Transformer based models for DU dating from late 2019. However, the computational complexity of the self-attention operation limits their capabilities to small sequences. In this paper we explore multiple strategies to apply Transformer based models to long multi-page documents. We introduce 2 new multi-modal (text + layout) long-range models for DU. They are based on efficient implementations of Transformers for long sequences. Long-range models can process whole documents at once effectively and are less impaired by the document's length. We compare them to LayoutLM, a classical Transformer adapted for DU and pre-trained on millions of documents. We further propose 2D relative attention bias to guide self-attention towards relevant tokens without harming model efficiency. We observe improvements on multi-page business documents on Information Retrieval for a small performance cost on smaller sequences. Relative 2D attention revealed to be effective on dense text for both normal and long-range models.