ToLeaP: Rethinking Development of Tool Learning with Large Language Models
This work addresses tool learning challenges for AI researchers and developers, but it is incremental as it builds on existing benchmarks and proposes directions rather than introducing a new method.
The paper tackles the problem of evaluating and improving tool learning in large language models by creating ToLeaP, a platform that reproduces 33 benchmarks and analyzes over 3,000 bad cases from 41 LLMs, identifying four key challenges such as benchmark limitations and lack of generalization.
Tool learning, which enables large language models (LLMs) to utilize external tools effectively, has garnered increasing attention for its potential to revolutionize productivity across industries. Despite rapid development in tool learning, key challenges and opportunities remain understudied, limiting deeper insights and future advancements. In this paper, we investigate the tool learning ability of 41 prevalent LLMs by reproducing 33 benchmarks and enabling one-click evaluation for seven of them, forming a Tool Learning Platform named ToLeaP. We also collect 21 out of 33 potential training datasets to facilitate future exploration. After analyzing over 3,000 bad cases of 41 LLMs based on ToLeaP, we identify four main critical challenges: (1) benchmark limitations induce both the neglect and lack of (2) autonomous learning, (3) generalization, and (4) long-horizon task-solving capabilities of LLMs. To aid future advancements, we take a step further toward exploring potential directions, namely (1) real-world benchmark construction, (2) compatibility-aware autonomous learning, (3) rationale learning by thinking, and (4) identifying and recalling key clues. The preliminary experiments demonstrate their effectiveness, highlighting the need for further research and exploration.