Towards AI Search Paradigm
This work addresses the problem of improving search systems for users by providing a comprehensive blueprint, though it appears incremental as it builds on existing LLM and agent-based approaches.
The paper tackles the challenge of creating next-generation search systems by proposing the AI Search Paradigm, a modular architecture with four LLM-powered agents that dynamically adapt to various information needs, from simple queries to complex reasoning tasks, aiming to inform the development of trustworthy, adaptive, and scalable AI search systems.
In this paper, we introduce the AI Search Paradigm, a comprehensive blueprint for next-generation search systems capable of emulating human information processing and decision-making. The paradigm employs a modular architecture of four LLM-powered agents (Master, Planner, Executor and Writer) that dynamically adapt to the full spectrum of information needs, from simple factual queries to complex multi-stage reasoning tasks. These agents collaborate dynamically through coordinated workflows to evaluate query complexity, decompose problems into executable plans, and orchestrate tool usage, task execution, and content synthesis. We systematically present key methodologies for realizing this paradigm, including task planning and tool integration, execution strategies, aligned and robust retrieval-augmented generation, and efficient LLM inference, spanning both algorithmic techniques and infrastructure-level optimizations. By providing an in-depth guide to these foundational components, this work aims to inform the development of trustworthy, adaptive, and scalable AI search systems.