Joint Continual Learning of Local Language Models and Cloud Offloading Decisions with Budget Constraints
This addresses the problem of efficient and stable cloud offloading for SLMs in resource-constrained environments, representing an incremental improvement over existing collaborative and routing-based approaches.
The paper tackles the challenge of regulating cloud assistance for Small Language Models (SLMs) during continual learning under budget constraints, proposing DA-GRPO, which improves post-switch accuracy, reduces forgetting, and maintains stable cloud usage compared to prior methods.
Locally deployed Small Language Models (SLMs) must continually support diverse tasks under strict memory and computation constraints, making selective reliance on cloud Large Language Models (LLMs) unavoidable. Regulating cloud assistance during continual learning is challenging, as naive reward-based reinforcement learning often yields unstable offloading behavior and exacerbates catastrophic forgetting as task distributions shift. We propose DA-GRPO, a dual-advantage extension of Group Relative Policy Optimization that incorporates cloud-usage constraints directly into advantage computation, avoiding fixed reward shaping and external routing models. This design enables the local model to jointly learn task competence and collaboration behavior, allowing cloud requests to emerge naturally during post-training while respecting a prescribed assistance budget. Experiments on mathematical reasoning and code generation benchmarks show that DA-GRPO improves post-switch accuracy, substantially reduces forgetting, and maintains stable cloud usage compared to prior collaborative and routing-based approaches.