TGPO: Tree-Guided Preference Optimization for Robust Web Agent Reinforcement Learning
This work addresses critical issues in web agent reinforcement learning for automated web interaction, representing an incremental improvement with novel method components.
The paper tackled challenges in training web agents with reinforcement learning, such as credit assignment misallocation and reward sparsity, by proposing Tree-Guided Preference Optimization (TGPO), which achieved higher success rates with fewer redundant steps on datasets like Online-Mind2Web and C-WebShop.
With the rapid advancement of large language models and vision-language models, employing large models as Web Agents has become essential for automated web interaction. However, training Web Agents with reinforcement learning faces critical challenges including credit assignment misallocation, prohibitively high annotation costs, and reward sparsity. To address these issues, we propose Tree-Guided Preference Optimization (TGPO), an offline reinforcement learning framework that proposes a tree-structured trajectory representation merging semantically identical states across trajectories to eliminate label conflicts. Our framework incorporates a Process Reward Model that automatically generates fine-grained rewards through subgoal progress, redundancy detection, and action verification. Additionally, a dynamic weighting mechanism prioritizes high-impact decision points during training. Experiments on Online-Mind2Web and our self-constructed C-WebShop datasets demonstrate that TGPO significantly outperforms existing methods, achieving higher success rates with fewer redundant steps.