Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindi
This work addresses the problem of limited Hindi language models for researchers and practitioners, though it is incremental as it builds upon existing Llama-3 and Llama Pro methodologies.
The authors tackled the challenge of developing high-quality large language models for Hindi, a moderately resourced language, by introducing Llama-3-Nanda-10B-Chat, which achieves state-of-the-art results as a top-performing open-source Hindi-centric model with 10 billion parameters.
Developing high-quality large language models (LLMs) for moderately resourced languages presents unique challenges in data availability, model adaptation, and evaluation. We introduce Llama-3-Nanda-10B-Chat, or Nanda for short, a state-of-the-art Hindi-centric instruction-tuned generative LLM, designed to push the boundaries of open-source Hindi language models. Built upon Llama-3-8B, Nanda incorporates continuous pre-training with expanded transformer blocks, leveraging the Llama Pro methodology. A key challenge was the limited availability of high-quality Hindi text data; we addressed this through rigorous data curation, augmentation, and strategic bilingual training, balancing Hindi and English corpora to optimize cross-linguistic knowledge transfer. With 10 billion parameters, Nanda stands among the top-performing open-source Hindi and multilingual models of similar scale, demonstrating significant advantages over many existing models. We provide an in-depth discussion of training strategies, fine-tuning techniques, safety alignment, and evaluation metrics, demonstrating how these approaches enabled Nanda to achieve state-of-the-art results. By open-sourcing Nanda, we aim to advance research in Hindi LLMs and support a wide range of real-world applications across academia, industry, and public services.