Extending the Pre-Training of BLOOM for Improved Support of Traditional Chinese: Models, Methods and Results
This work addresses the problem of limited Traditional Chinese support in multilingual models for researchers and users in that domain, but it is incremental as it builds on an existing model.
The paper tackled improving support for Traditional Chinese in the BLOOM language model by extending pre-training with 7.4 billion tokens, resulting in BLOOM-zh outperforming its predecessor on most Traditional Chinese benchmarks while maintaining English capability.
In this paper we present the multilingual language model BLOOM-zh that features enhanced support for Traditional Chinese. BLOOM-zh has its origins in the open-source BLOOM models presented by BigScience in 2022. Starting from released models, we extended the pre-training of BLOOM by additional 7.4 billion tokens in Traditional Chinese and English covering a variety of domains such as news articles, books, encyclopedias, educational materials as well as spoken language. In order to show the properties of BLOOM-zh, both existing and newly created benchmark scenarios are used for evaluating the performance. BLOOM-zh outperforms its predecessor on most Traditional Chinese benchmarks while maintaining its English capability. We release all our models to the research community.