Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimization
For practitioners deploying tool-integrated agents, this addresses the overlooked trade-off between accuracy and efficiency, enabling more practical alignment.
ParetoPO introduces a multi-objective optimization framework for aligning tool-using LLMs, achieving superior accuracy-efficiency trade-offs on math reasoning and multi-hop QA tasks compared to static baselines.
Recent advances in tool-integrated language agents have significantly improved their ability to solve complex reasoning tasks. However, existing alignment methods predominantly focus on maximizing task accuracy, while overlooking auxiliary objectives such as tool-use efficiency, which are essential for practical deployment. To address this gap, we introduce ParetoPO, a two-stage multi-objective optimization framework for aligning tool-using large language models (LLMs) under competing objectives. In the first stage, ParetoPO leverages hypervolume-guided dynamic scalarization to adapt reward weights based on global Pareto frontier progress. In the second stage, it replaces scalarized learning signals with Pareto-ranking-based advantage computation, promoting nondominated trajectories through dominance-aware credit assignment. This design enables fine-grained, action-level optimization across multiple conflicting objectives. Experimental results on mathematic reasoning and multi-hop QA tasks show that ParetoPO consistently discovers policies with superior accuracy-efficiency trade-offs compared to static and heuristic baselines.