Adaptive Multimodal Agents-Based Framework for Automatic Workflow Execution
For developers of autonomous agents in information systems, this framework addresses the limitation of treating task sequences as discrete episodes by capturing transition topology, but the improvement is incremental.
The paper proposes a multimodal multi-agent framework for automatic workflow execution that constructs a topological knowledge base from execution logs and uses adaptive RAG with closed-loop verification to navigate workflows. It achieves high reliability and semantic awareness with limited training data.
Modern information systems require autonomous agents capable of navigating complex workflows, yet current methodologies often struggle with the transition from structured metadata parsing to general environmental perception. While the integration of MLLMs has enabled agents to interact directly with GUIs, existing approaches typically treat task sequences as discrete, linear episodes. This fragmentation prevents agents from capturing the underlying transition topology, limiting their effectiveness in novel or non-stationary scenarios. To address this, we propose a novel multimodal multi-agent framework that achieves automatic workflow execution through a distinct two-phase pipeline. First, during an offline discovery phase, the architecture adaptively constructs a topological knowledge base from fragmented execution logs. During inference, agents leverage Adaptive Retrieval-Augmented Generation (RAG) over this fixed, pre-established graph, coupled with a closed-loop collaborative verification protocol to dynamically self-correct and navigate. This graph-based approach facilitates superior task decomposition and adaptive navigation performance. We validate our framework in a real-world context, demonstrating its ability to maintain high reliability and semantic awareness even with limited training data.