Open Artificial Knowledge
This addresses data scarcity and privacy issues in LLM training for AI researchers and developers, though it is incremental as it builds on existing methods for data generation.
The paper tackles the challenge of acquiring high-quality, diverse, and ethically sourced training data for large language models by introducing the Open Artificial Knowledge (OAK) dataset, a large-scale resource of over 500 million tokens generated using an ensemble of state-of-the-art LLMs to ensure broad knowledge coverage and factual accuracy.
The tremendous success of chat-based AI systems like ChatGPT, Claude, and Gemini stems from Large Language Models (LLMs) trained on vast amount of datasets. However, acquiring high-quality, diverse, and ethically sourced training data remains a significant challenge. We introduce the Open Artificial Knowledge (OAK) dataset, a large-scale resource of over 500 million tokens (at the moment of writing) designed to address this issue. OAK leverages an ensemble of state-of-the-art LLMs, including GPT4o, LLaMa3-70B, LLaMa3-8B, Mixtral-8x7B, Gemma-7B, and Gemma-2-9B , to generate high-quality text across diverse domains, guided by Wikipedia's main categories. Our methodology ensures broad knowledge coverage while maintaining coherence and factual accuracy. The OAK dataset aims to foster the development of more capable and aligned language models while addressing critical issues of data scarcity and privacy in LLM training, and it is freely available on www.oakdataset.org.