Efficient Extractive Summarization with MAMBA-Transformer Hybrids for Low-Resource Scenarios
This addresses the bottleneck of quadratic complexity in summarization for resource-constrained scenarios, though it is incremental as it builds on existing transformer and state space models.
The paper tackled the problem of extractive summarization for long documents in low-resource settings by introducing a Mamba-Transformer hybrid, achieving gains such as +0.23 ROUGE-1 on ArXiv and 24-27% faster inference on news summarization.
Extractive summarization of long documents is bottlenecked by quadratic complexity, often forcing truncation and limiting deployment in resource-constrained settings. We introduce the first Mamba-Transformer hybrid for extractive summarization, combining the semantic strength of pre-trained transformers with the linear-time processing of state space models. Leveraging Mamba's ability to process full documents without truncation, our approach preserves context while maintaining strong summarization quality. The architecture includes: (1) a transformer encoder for sentence-level semantics, (2) a Mamba state space model to capture inter-sentence dependencies efficiently, and (3) a linear classifier for sentence relevance prediction. Across news, argumentative, and scientific domains under low-resource conditions, our method achieves: (1) large gains over BERTSUM and MATCHSUM, including +0.23 ROUGE-1 on ArXiv and statistically significant improvements on all datasets (p < 0.001); (2) consistent advantages across domains, strongest on the longest documents; (3) robust performance with limited training data; and (4) 24-27% faster inference on news summarization (CNN/DailyMail). We introduce the first hybrid Transformer-state space architecture for summarization, showing significant ROUGE improvements in low-resource scenarios.