TravelBench : Exploring LLM Performance in Low-Resource Domains
This work addresses the challenge of evaluating LLMs in low-resource domains like travel, which is incremental as it focuses on benchmarking rather than new methods.
The authors tackled the problem of insufficient information from existing LLM benchmarks for low-resource tasks by curating 14 travel-domain datasets across 7 NLP tasks and analyzing LLM performance, finding that general benchmarks are inadequate and out-of-the-box LLMs face bottlenecks in domain-specific scenarios, with reasoning boosting smaller models.
Results on existing LLM benchmarks capture little information over the model capabilities in low-resource tasks, making it difficult to develop effective solutions in these domains. To address these challenges, we curated 14 travel-domain datasets spanning 7 common NLP tasks using anonymised data from real-world scenarios, and analysed the performance across LLMs. We report on the accuracy, scaling behaviour, and reasoning capabilities of LLMs in a variety of tasks. Our results confirm that general benchmarking results are insufficient for understanding model performance in low-resource tasks. Despite the amount of training FLOPs, out-of-the-box LLMs hit performance bottlenecks in complex, domain-specific scenarios. Furthermore, reasoning provides a more significant boost for smaller LLMs by making the model a better judge on certain tasks.