Long-Context Encoder Models for Polish Language Understanding
This addresses the problem of processing long documents for Polish language understanding, offering a cost-effective solution for discriminative tasks, though it is incremental as it builds on existing encoder architectures.
The authors tackled the short context window limitation of encoder models for Polish language by developing a model that processes up to 8192 tokens, achieving the best average performance on 25 tasks, including long-document understanding, while maintaining quality on short texts.
While decoder-only Large Language Models (LLMs) have recently dominated the NLP landscape, encoder-only architectures remain a cost-effective and parameter-efficient standard for discriminative tasks. However, classic encoders like BERT are limited by a short context window, which is insufficient for processing long documents. In this paper, we address this limitation for the Polish by introducing a high-quality Polish model capable of processing sequences of up to 8192 tokens. The model was developed by employing a two-stage training procedure that involves positional embedding adaptation and full parameter continuous pre-training. Furthermore, we propose compressed model variants trained via knowledge distillation. The models were evaluated on 25 tasks, including the KLEJ benchmark, a newly introduced financial task suite (FinBench), and other classification and regression tasks, specifically those requiring long-document understanding. The results demonstrate that our model achieves the best average performance among Polish and multilingual models, significantly outperforming competitive solutions in long-context tasks while maintaining comparable quality on short texts.