Skim-Attention: Learning to Focus via Document Layout
This addresses efficiency issues in document AI for researchers and practitioners, though it is incremental as it builds on existing attention mechanisms.
The paper tackles the high computational and memory costs of multimodal pre-training models for document understanding by introducing Skim-Attention, an attention mechanism that uses document layout to focus on word positions, resulting in lower perplexity and improved efficiency.
Transformer-based pre-training techniques of text and layout have proven effective in a number of document understanding tasks. Despite this success, multimodal pre-training models suffer from very high computational and memory costs. Motivated by human reading strategies, this paper presents Skim-Attention, a new attention mechanism that takes advantage of the structure of the document and its layout. Skim-Attention only attends to the 2-dimensional position of the words in a document. Our experiments show that Skim-Attention obtains a lower perplexity than prior works, while being more computationally efficient. Skim-Attention can be further combined with long-range Transformers to efficiently process long documents. We also show how Skim-Attention can be used off-the-shelf as a mask for any Pre-trained Language Model, allowing to improve their performance while restricting attention. Finally, we show the emergence of a document structure representation in Skim-Attention.