Can LLMs Outshine Conventional Recommenders? A Comparative Evaluation
This work addresses the need for a systematic comparison of LLMs and traditional recommenders for researchers and practitioners, though it is incremental as it focuses on benchmarking rather than proposing new methods.
The paper tackles the problem of evaluating large language models (LLMs) against conventional recommender systems by introducing RecBench, a comprehensive benchmark covering various item representations and tasks, finding that LLM-based recommenders outperform traditional ones with up to a 5% AUC improvement in click-through rate prediction and up to a 170% NDCG@10 improvement in sequential recommendation, but at the cost of reduced inference efficiency.
In recent years, integrating large language models (LLMs) into recommender systems has created new opportunities for improving recommendation quality. However, a comprehensive benchmark is needed to thoroughly evaluate and compare the recommendation capabilities of LLMs with traditional recommender systems. In this paper, we introduce RecBench, which systematically investigates various item representation forms (including unique identifier, text, semantic embedding, and semantic identifier) and evaluates two primary recommendation tasks, i.e., click-through rate prediction (CTR) and sequential recommendation (SeqRec). Our extensive experiments cover up to 17 large models and are conducted across five diverse datasets from fashion, news, video, books, and music domains. Our findings indicate that LLM-based recommenders outperform conventional recommenders, achieving up to a 5% AUC improvement in the CTR scenario and up to a 170% NDCG@10 improvement in the SeqRec scenario. However, these substantial performance gains come at the expense of significantly reduced inference efficiency, rendering the LLM-as-RS paradigm impractical for real-time recommendation environments. We aim for our findings to inspire future research, including recommendation-specific model acceleration methods. We will release our code, data, configurations, and platform to enable other researchers to reproduce and build upon our experimental results.