ToolSpectrum : Towards Personalized Tool Utilization for Large Language Models
This work addresses the need for context-aware personalization in tool-augmented LLMs, which is incremental as it builds on existing tool integration approaches by adding a personalization dimension.
The paper tackles the problem of suboptimal tool selection in large language models by introducing ToolSpectrum, a benchmark for evaluating personalized tool utilization based on user profiles and environmental factors, showing that personalization improves user experience but current models struggle with joint reasoning.
While integrating external tools into large language models (LLMs) enhances their ability to access real-time information and domain-specific services, existing approaches focus narrowly on functional tool selection following user instructions, overlooking the context-aware personalization in tool selection. This oversight leads to suboptimal user satisfaction and inefficient tool utilization, particularly when overlapping toolsets require nuanced selection based on contextual factors. To bridge this gap, we introduce ToolSpectrum, a benchmark designed to evaluate LLMs' capabilities in personalized tool utilization. Specifically, we formalize two key dimensions of personalization, user profile and environmental factors, and analyze their individual and synergistic impacts on tool utilization. Through extensive experiments on ToolSpectrum, we demonstrate that personalized tool utilization significantly improves user experience across diverse scenarios. However, even state-of-the-art LLMs exhibit the limited ability to reason jointly about user profiles and environmental factors, often prioritizing one dimension at the expense of the other. Our findings underscore the necessity of context-aware personalization in tool-augmented LLMs and reveal critical limitations for current models. Our data and code are available at https://github.com/Chengziha0/ToolSpectrum.