Embedded AI Companion System on Edge Devices
This work addresses the problem of enabling AI companions on resource-constrained edge devices for users needing low-latency, personalized interactions, though it is incremental as it builds on existing memory and retrieval methods.
The paper tackles the challenge of embedding an AI companion system on edge devices with limited computational resources by proposing a memory paradigm that alternates between active and inactive phases to minimize latency while maintaining long-term personalization. The system, using a quantized Qwen2.5-7B-Instruct model, outperforms an equivalent raw LLM without memory in most metrics and performs comparably to GPT-3.5 with a 16k context window.
Computational resource constraints on edge devices make it difficult to develop a fully embedded AI companion system with a satisfactory user experience. AI companion and memory systems detailed in existing literature cannot be directly used in such an environment due to lack of compute resources and latency concerns. In this paper, we propose a memory paradigm that alternates between active and inactive phases: during phases of user activity, the system performs low-latency, real-time dialog using lightweight retrieval over existing memories and context; whereas during phases of user inactivity, it conducts more computationally intensive extraction, consolidation, and maintenance of memories across full conversation sessions. This design minimizes latency while maintaining long-term personalization under the tight constraints of embedded hardware. We also introduce an AI Companion benchmark designed to holistically evaluate the AI Companion across both its conversational quality and memory capabilities. In our experiments, we found that our system (using a very weak model: Qwen2.5-7B-Instruct quantized int4) outperforms the equivalent raw LLM without memory across most metrics, and performs comparably to GPT-3.5 with 16k context window.