Every Software as an Agent: Blueprint and Case Study
This work addresses the challenge of creating more effective software agents for users by introducing a new paradigm that could revolutionize agent design, though it is presented as a blueprint with early-stage case studies.
The paper tackles the problem of low accuracy and efficiency in existing software agents by proposing a whitebox approach that allows LLMs to access software internals and inject generated code, demonstrating its potential through case studies on two web-based desktop applications.
The rise of (multimodal) large language models (LLMs) has shed light on software agent -- where software can understand and follow user instructions in natural language. However, existing approaches such as API-based and GUI-based agents are far from satisfactory at accuracy and efficiency aspects. Instead, we advocate to endow LLMs with access to the software internals (source code and runtime context) and the permission to dynamically inject generated code into software for execution. In such a whitebox setting, one may better leverage the software context and the coding ability of LLMs. We then present an overall design architecture and case studies on two popular web-based desktop applications. We also give in-depth discussion of the challenges and future directions. We deem that such a new paradigm has the potential to fundamentally overturn the existing software agent design, and finally creating a digital world in which software can comprehend, operate, collaborate, and even think to meet complex user needs.