Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executors
This provides an efficient alternative to fine-tuning for researchers and practitioners using LLMs, though it is incremental as it builds on existing workflow optimization methods.
The paper tackles the challenge of efficiently leveraging large language models (LLMs) without expensive fine-tuning by proposing the Weak-for-Strong Harnessing (W4S) framework, which trains a smaller 7B meta-agent to design workflows that improve the performance of stronger models like GPT-3.5-Turbo and GPT-4o by 2.9% to 24.6% across eleven benchmarks using only one GPU hour.
Efficiently leveraging of the capabilities of contemporary large language models (LLMs) is increasingly challenging, particularly when direct fine-tuning is expensive and often impractical. Existing training-free methods, including manually or automated designed workflows, typically demand substantial human effort or yield suboptimal results. This paper proposes Weak-for-Strong Harnessing (W4S), a novel framework that customizes smaller, cost-efficient language models to design and optimize workflows for harnessing stronger models. W4S formulates workflow design as a multi-turn markov decision process and introduces reinforcement learning for agentic workflow optimization (RLAO) to train a weak meta-agent. Through iterative interaction with the environment, the meta-agent learns to design increasingly effective workflows without manual intervention. Empirical results demonstrate the superiority of W4S that our 7B meta-agent, trained with just one GPU hour, outperforms the strongest baseline by 2.9% ~ 24.6% across eleven benchmarks, successfully elevating the performance of state-of-the-art models such as GPT-3.5-Turbo and GPT-4o. Notably, W4S exhibits strong generalization capabilities across both seen and unseen tasks, offering an efficient, high-performing alternative to directly fine-tuning strong models.