PhoGPT: Generative Pre-training for Vietnamese
This provides a high-quality generative AI tool for Vietnamese speakers, though it is incremental as it adapts existing methods to a new language.
The authors tackled the lack of a state-of-the-art generative model for Vietnamese by developing PhoGPT, a 4B-parameter series including a base model pre-trained on 102B tokens and a chat variant fine-tuned on 360K prompts, achieving superior performance over previous open-source models.
We open-source a state-of-the-art 4B-parameter generative model series for Vietnamese, which includes the base pre-trained monolingual model PhoGPT-4B and its chat variant, PhoGPT-4B-Chat. The base model, PhoGPT-4B, with exactly 3.7B parameters, is pre-trained from scratch on a Vietnamese corpus of 102B tokens, with an 8192 context length, employing a vocabulary of 20480 token types. The chat variant, PhoGPT-4B-Chat, is the modeling output obtained by fine-tuning PhoGPT-4B on a dataset of 70K instructional prompts and their responses, along with an additional 290K conversations. In addition, we also demonstrate its superior performance compared to previous open-source models. Our PhoGPT models are available at: https://github.com/VinAIResearch/PhoGPT