Smaller, Smarter, Closer: The Edge of Collaborative Generative AI
This work addresses deployment challenges for generative AI in edge computing environments, providing incremental guidance for system design.
The paper tackles the limitations of cloud-centric generative AI deployments, such as latency and privacy, by exploring collaborative inference systems that combine edge and cloud resources, offering practical design principles and experimental insights for deployment.
The rapid adoption of generative AI (GenAI), particularly Large Language Models (LLMs), has exposed critical limitations of cloud-centric deployments, including latency, cost, and privacy concerns. Meanwhile, Small Language Models (SLMs) are emerging as viable alternatives for resource-constrained edge environments, though they often lack the capabilities of their larger counterparts. This article explores the potential of collaborative inference systems that leverage both edge and cloud resources to address these challenges. By presenting distinct cooperation strategies alongside practical design principles and experimental insights, we offer actionable guidance for deploying GenAI across the computing continuum.