KoBigBird-large: Transformation of Transformer for Korean Language Understanding
This work addresses Korean language processing, offering improved performance for tasks like document classification and question answering, but it is incremental as it builds on existing BigBird architecture.
The authors tackled Korean language understanding by developing KoBigBird-large, a model that achieved state-of-the-art performance on benchmarks, including the best results on document classification and question answering for longer sequences.
This work presents KoBigBird-large, a large size of Korean BigBird that achieves state-of-the-art performance and allows long sequence processing for Korean language understanding. Without further pretraining, we only transform the architecture and extend the positional encoding with our proposed Tapered Absolute Positional Encoding Representations (TAPER). In experiments, KoBigBird-large shows state-of-the-art overall performance on Korean language understanding benchmarks and the best performance on document classification and question answering tasks for longer sequences against the competitive baseline models. We publicly release our model here.