Efficient Zero-Shot Long Document Classification by Reducing Context Through Sentence Ranking
This addresses computational inefficiencies in LDC for applications like news classification, though it is incremental as it builds on existing methods without altering model architecture.
The paper tackles the problem of long document classification (LDC) with transformer models by proposing a zero-shot approach that uses sentence ranking to reduce input context, achieving up to 35% faster inference while maintaining comparable accuracy on the MahaNews dataset.
Transformer-based models like BERT excel at short text classification but struggle with long document classification (LDC) due to input length limitations and computational inefficiencies. In this work, we propose an efficient, zero-shot approach to LDC that leverages sentence ranking to reduce input context without altering the model architecture. Our method enables the adaptation of models trained on short texts, such as headlines, to long-form documents by selecting the most informative sentences using a TF-IDF-based ranking strategy. Using the MahaNews dataset of long Marathi news articles, we evaluate three context reduction strategies that prioritize essential content while preserving classification accuracy. Our results show that retaining only the top 50\% ranked sentences maintains performance comparable to full-document inference while reducing inference time by up to 35\%. This demonstrates that sentence ranking is a simple yet effective technique for scalable and efficient zero-shot LDC.