Mini-GPTs: Efficient Large Language Models through Contextual Pruning
This work addresses the need for resource-efficient AI models for practical applications, though it is incremental, building on existing pruning techniques.
The paper tackles the challenge of optimizing large language models by introducing contextual pruning to reduce model sizes while retaining core functionalities, achieving efficient domain-specific LLMs across diverse datasets like US law and Medical Q&A.
In AI research, the optimization of Large Language Models (LLMs) remains a significant challenge, crucial for advancing the field's practical applications and sustainability. Building upon the foundational work of Professor Song Han's lab at MIT, this paper introduces a novel approach in developing Mini-GPTs via contextual pruning. Our methodology strategically prunes the computational architecture of traditional LLMs, like Phi-1.5, focusing on retaining core functionalities while drastically reducing model sizes. We employ the technique across diverse and complex datasets, including US law, Medical Q&A, Skyrim dialogue, English-Taiwanese translation, and Economics articles. The results underscore the efficiency and effectiveness of contextual pruning, not merely as a theoretical concept but as a practical tool in developing domain-specific, resource-efficient LLMs. Contextual pruning is a promising method for building domain-specific LLMs, and this research is a building block towards future development with more hardware compute, refined fine-tuning, and quantization.