DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models
This addresses the challenge of making TA-LLMs more robust in real-world applications by improving their ability to handle diverse user queries without needing expert demonstrations or human labeling, representing a strong specific gain in this domain.
The paper tackled the problem of Tool-Augmented Large Language Models (TA-LLMs) struggling with incomplete queries and out-of-scope requests by proposing DiaTool-DPO, a method using Direct Preference Optimization to enhance dialogue capabilities, resulting in performance approaching GPT-4o (e.g., 94.8% in information gathering) with substantial improvements over baselines (e.g., 44% to 94.8%).
Tool-Augmented Larage Language Models (TA-LLMs) have shown promise in real-world applications, but face challenges in handling incomplete queries and out-of-scope requests. While existing approaches rely mainly on Supervised Fine-Tuning with expert trajectories, we propose DiaTool-DPO, a novel method that enhances TA-LLM's dialogue capabilities through Direct Preference Optimization. We model TA-LLM interactions as a Markov Decision Process with 5 distinct dialogue states and categorize user queries into 3 types based on their state transition trajectories. We automatically construct paired trajectory datasets of correct and incorrect dialogue flows and introduce a specialized objective loss for dialogue control. Our comprehensive evaluation demonstrates that DiaTool-DPO approaches GPT-4o's performance (94.8% in information gathering, 91% in tool call rejection) with substantial improvements over baseline (44% and 9.6% respectively) while maintaining core functionality. Our approach opens new possibilities for developing TA-LLMs that can handle diverse real-world scenarios without requiring additional expert demonstrations or human labeling.