ScaleFormer: Span Representation Cumulation for Long-Context Transformer
This addresses the problem of efficiently handling long sequences for NLP practitioners, offering an incremental improvement by enabling existing models to work on long-form text without costly retraining.
The paper tackles the quadratic complexity of standard self-attention in Transformers for long-context tasks by proposing ScaleFormer, a plug-and-play framework that adapts pre-trained models to process long sequences without architectural changes, achieving competitive or superior performance in long-document summarization experiments.
The quadratic complexity of standard self-attention severely limits the application of Transformer-based models to long-context tasks. While efficient Transformer variants exist, they often require architectural changes and costly pre-training from scratch. To circumvent this, we propose ScaleFormer(Span Representation Cumulation for Long-Context Transformer) - a simple and effective plug-and-play framework that adapts off-the-shelf pre-trained encoder-decoder models to process long sequences without requiring architectural modifications. Our approach segments long inputs into overlapping chunks and generates a compressed, context-aware representation for the decoder. The core of our method is a novel, parameter-free fusion mechanism that endows each chunk's representation with structural awareness of its position within the document. It achieves this by enriching each chunk's boundary representations with cumulative context vectors from all preceding and succeeding chunks. This strategy provides the model with a strong signal of the document's narrative flow, achieves linear complexity, and enables pre-trained models to reason effectively over long-form text. Experiments on long-document summarization show that our method is highly competitive with and often outperforms state-of-the-art approaches without requiring architectural modifications or external retrieval mechanisms.