An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification
This work addresses efficiency challenges in long document processing for NLP practitioners, though it is incremental as it builds on existing hierarchical attention methods.
The authors tackled the problem of efficient long document classification by developing fully pre-trained Hierarchical Attention Transformer (HAT) models, which outperformed equally-sized Longformer models by using 10-20% less GPU memory and processing documents 40-45% faster in downstream tasks.
Non-hierarchical sparse attention Transformer-based models, such as Longformer and Big Bird, are popular approaches to working with long documents. There are clear benefits to these approaches compared to the original Transformer in terms of efficiency, but Hierarchical Attention Transformer (HAT) models are a vastly understudied alternative. We develop and release fully pre-trained HAT models that use segment-wise followed by cross-segment encoders and compare them with Longformer models and partially pre-trained HATs. In several long document downstream classification tasks, our best HAT model outperforms equally-sized Longformer models while using 10-20% less GPU memory and processing documents 40-45% faster. In a series of ablation studies, we find that HATs perform best with cross-segment contextualization throughout the model than alternative configurations that implement either early or late cross-segment contextualization. Our code is on GitHub: https://github.com/coastalcph/hierarchical-transformers.