Position: Enough of Scaling LLMs! Lets Focus on Downscaling
It addresses the problem of unsustainable and inefficient LLM development for researchers and practitioners, advocating for a more accessible approach, though it is incremental as it builds on existing scaling critiques.
The paper challenges the focus on scaling laws for large language models (LLMs) and advocates for a paradigm shift toward downscaling to address computational inefficiency, environmental impact, and deployment constraints, proposing a holistic framework to maintain performance while reducing resource demands.
We challenge the dominant focus on neural scaling laws and advocate for a paradigm shift toward downscaling in the development of large language models (LLMs). While scaling laws have provided critical insights into performance improvements through increasing model and dataset size, we emphasize the significant limitations of this approach, particularly in terms of computational inefficiency, environmental impact, and deployment constraints. To address these challenges, we propose a holistic framework for downscaling LLMs that seeks to maintain performance while drastically reducing resource demands. This paper outlines practical strategies for transitioning away from traditional scaling paradigms, advocating for a more sustainable, efficient, and accessible approach to LLM development.