XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters
This addresses the need for specialized chat models in Chinese finance, though it is incremental as it builds on existing architectures.
The authors tackled the lack of open-sourced large-scale Chinese financial chat models by introducing XuanYuan 2.0, a model with hundreds of billions of parameters based on BLOOM-176B, which provides accurate responses in the Chinese financial domain.
In recent years, pre-trained language models have undergone rapid development with the emergence of large-scale models. However, there is a lack of open-sourced chat models specifically designed for the Chinese language, especially in the field of Chinese finance, at the scale of hundreds of billions. To address this gap, we introduce XuanYuan 2.0, the largest Chinese chat model to date, built upon the BLOOM-176B architecture. Additionally, we propose a novel training method called hybrid-tuning to mitigate catastrophic forgetting. By combining general-domain with domain-specific knowledge and integrating the stages of pre-training and fine-tuning, XuanYuan 2.0 is capable of providing accurate and contextually appropriate responses in the Chinese financial domain.