Salespeople vs SalesBot: Exploring the Role of Educational Value in Conversational Recommender Systems
This work addresses the challenge of enhancing educational value in conversational recommender systems for consumers making big purchases, representing an incremental advancement by applying LLMs to a new problem space.
The paper tackles the problem of conversational recommender systems overlooking users' lack of domain expertise by defining a new problem space for agents that provide both product recommendations and educational value, and introduces SalesOps, a framework for simulating and evaluating such systems using LLMs, with a human study showing SalesBot approaches professional salespeople in fluency and informativeness but lags in recommendation quality.
Making big purchases requires consumers to research or consult a salesperson to gain domain expertise. However, existing conversational recommender systems (CRS) often overlook users' lack of background knowledge, focusing solely on gathering preferences. In this work, we define a new problem space for conversational agents that aim to provide both product recommendations and educational value through mixed-type mixed-initiative dialog. We introduce SalesOps, a framework that facilitates the simulation and evaluation of such systems by leveraging recent advancements in large language models (LLMs). We build SalesBot and ShopperBot, a pair of LLM-powered agents that can simulate either side of the framework. A comprehensive human study compares SalesBot against professional salespeople, revealing that although SalesBot approaches professional performance in terms of fluency and informativeness, it lags behind in recommendation quality. We emphasize the distinct limitations both face in providing truthful information, highlighting the challenges of ensuring faithfulness in the CRS context. We release our code and make all data available.