Octopus v2: On-device language model for super agent
This enables efficient, private, and cost-effective AI agents on edge devices, addressing a bottleneck in real-world deployment.
The authors tackled the problem of high latency and low accuracy in on-device language models for function calling by developing a 2-billion-parameter model that surpasses GPT-4 in accuracy and latency, reduces context length by 95%, and improves latency by 35-fold compared to Llama-7B with RAG.
Language models have shown effectiveness in a variety of software applications, particularly in tasks related to automatic workflow. These models possess the crucial ability to call functions, which is essential in creating AI agents. Despite the high performance of large-scale language models in cloud environments, they are often associated with concerns over privacy and cost. Current on-device models for function calling face issues with latency and accuracy. Our research presents a new method that empowers an on-device model with 2 billion parameters to surpass the performance of GPT-4 in both accuracy and latency, and decrease the context length by 95\%. When compared to Llama-7B with a RAG-based function calling mechanism, our method enhances latency by 35-fold. This method reduces the latency to levels deemed suitable for deployment across a variety of edge devices in production environments, aligning with the performance requisites for real-world applications.