Ads that Talk Back: Implications and Perceptions of Injecting Personalized Advertising into LLM Chatbots
This addresses the profitability challenge for companies deploying LLMs by exploring ad-based monetization, though it is incremental in applying existing advertising concepts to a new platform.
The paper investigates the impact of injecting personalized advertisements into LLM chatbot responses, finding that ad injection slightly affects performance and that participants often fail to detect ads, even preferring responses with hidden advertisements.
Recent advances in large language models (LLMs) have enabled the creation of highly effective chatbots. However, the compute costs of widely deploying LLMs have raised questions about profitability. Companies have proposed exploring ad-based revenue streams for monetizing LLMs, which could serve as the new de facto platform for advertising. This paper investigates the implications of personalizing LLM advertisements to individual users via a between-subjects experiment with 179 participants. We developed a chatbot that embeds personalized product advertisements within LLM responses, inspired by similar forays by AI companies. The evaluation of our benchmarks showed that ad injection only slightly impacted LLM performance, particularly response desirability. Results revealed that participants struggled to detect ads, and even preferred LLM responses with hidden advertisements. Rather than clicking on our advertising disclosure, participants tried changing their advertising settings using natural language queries. We created an advertising dataset and an open-source LLM, Phi-4-Ads, fine-tuned to serve ads and flexibly adapt to user preferences.