LLMind: Orchestrating AI and IoT with LLM for Complex Task Execution
This work addresses the problem of complex task execution in IoT systems for users and developers, representing an incremental improvement by integrating LLMs with domain-specific modules.
The paper tackles the limitation of existing IoT systems in handling complex tasks by introducing LLMind, an LLM-based framework that orchestrates AI modules and IoT devices to execute tasks based on high-level human instructions, achieving enhanced capabilities through a novel experience accumulation mechanism.
Task-oriented communications are an important element in future intelligent IoT systems. Existing IoT systems, however, are limited in their capacity to handle complex tasks, particularly in their interactions with humans to accomplish these tasks. In this paper, we present LLMind, an LLM-based task-oriented AI agent framework that enables effective collaboration among IoT devices, with humans communicating high-level verbal instructions, to perform complex tasks. Inspired by the functional specialization theory of the brain, our framework integrates an LLM with domain-specific AI modules, enhancing its capabilities. Complex tasks, which may involve collaborations of multiple domain-specific AI modules and IoT devices, are executed through a control script generated by the LLM using a Language-Code transformation approach, which first converts language descriptions to an intermediate finite-state machine (FSM) before final precise transformation to code. Furthermore, the framework incorporates a novel experience accumulation mechanism to enhance response speed and effectiveness, allowing the framework to evolve and become progressively sophisticated through continuing user and machine interactions.