BanglaByT5: Byte-Level Modelling for Bangla
This addresses the need for effective NLP tools for Bangla, particularly in resource-constrained environments, though it is incremental as it adapts an existing architecture to a specific language.
The paper tackled the problem of large language models failing to capture nuances in morphologically rich languages like Bangla by introducing BanglaByT5, a byte-level encoder-decoder model tailored for Bangla, which demonstrated competitive performance in evaluations, surpassing several larger models.
Large language models (LLMs) have achieved remarkable success across various natural language processing tasks. However, most LLM models use traditional tokenizers like BPE and SentencePiece, which fail to capture the finer nuances of a morphologically rich language like Bangla (Bengali). In this work, we introduce BanglaByT5, the first byte-level encoder-decoder model explicitly tailored for Bangla. Built upon a small variant of Googles ByT5 architecture, BanglaByT5 is pre-trained on a 14GB curated corpus combining high-quality literary and newspaper articles. Through zeroshot and supervised evaluations across generative and classification tasks, BanglaByT5 demonstrates competitive performance, surpassing several multilingual and larger models. Our findings highlight the efficacy of byte-level modelling for morphologically rich languages and highlight BanglaByT5 potential as a lightweight yet powerful tool for Bangla NLP, particularly in both resource-constrained and scalable environments.