TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computers
This addresses the problem of latency, connectivity, and privacy for edge computing applications by enabling local inference, though it is incremental as it builds on existing model scaling trends.
The paper tackled the challenge of deploying large language models on edge devices by hypothesizing that smaller models (30-120M parameters) can outperform larger ones for specific tasks through careful data curation, and found that these small models achieve high token rates and accuracy when run locally on edge devices.
Language models have gained significant interest due to their general-purpose capabilities, which appear to emerge as models are scaled to increasingly larger parameter sizes. However, these large models impose stringent requirements on computing systems, necessitating significant memory and processing requirements for inference. This makes performing inference on mobile and edge devices challenging, often requiring invocating remotely-hosted models via network calls. Remote inference, in turn, introduces issues like latency, unreliable network connectivity, and privacy concerns. To address these challenges, we explored the possibility of deviating from the trend of increasing model size. Instead, we hypothesize that much smaller models (~30-120M parameters) can outperform their larger counterparts for specific tasks by carefully curating the data used for pre-training and fine-tuning. We investigate this within the context of deploying edge-device models to support sensing applications. We trained several foundational models through a systematic study and found that small models can run locally on edge devices, achieving high token rates and accuracy. Based on these findings, we developed a framework that allows users to train foundational models tailored to their specific applications and deploy them at the edge.