LP Data Pipeline: Lightweight, Purpose-driven Data Pipeline for Large Language Models
This addresses the accessibility issue for organizations lacking GPU infrastructure, enabling more efficient LLM development, though it is incremental as it builds on existing data curation methods.
The paper tackles the problem of resource-intensive dataset creation for large language models by introducing the LP Data Pipeline, a CPU-based framework that reduces preparation time and cost while maintaining high data quality.
Creating high-quality, large-scale datasets for large language models (LLMs) often relies on resource-intensive, GPU-accelerated models for quality filtering, making the process time-consuming and costly. This dependence on GPUs limits accessibility for organizations lacking significant computational infrastructure. To address this issue, we introduce the Lightweight, Purpose-driven (LP) Data Pipeline, a framework that operates entirely on CPUs to streamline the processes of dataset extraction, filtering, and curation. Based on our four core principles, the LP Data Pipeline significantly reduces preparation time and cost while maintaining high data quality. Importantly, our pipeline enables the creation of purpose-driven datasets tailored to specific domains and languages, enhancing the applicability of LLMs in specialized contexts. We anticipate that our pipeline will lower the barriers to LLM development, enabling a wide range of organizations to access LLMs more easily.