GEM-Bench: A Benchmark for Ad-Injected Response Generation within Generative Engine Marketing
This addresses a gap for researchers in GEM by providing a benchmark, but it is incremental as it builds on existing concepts without introducing a new paradigm.
The authors tackled the lack of benchmarks for ad-injected response generation in Generative Engine Marketing (GEM) by proposing GEM-Bench, a comprehensive benchmark with datasets, metrics, and baselines, finding that prompt-based methods boost engagement like click-through rates but reduce user satisfaction, while ad-insertion methods improve satisfaction at the cost of overhead.
Generative Engine Marketing (GEM) is an emerging ecosystem for monetizing generative engines, such as LLM-based chatbots, by seamlessly integrating relevant advertisements into their responses. At the core of GEM lies the generation and evaluation of ad-injected responses. However, existing benchmarks are not specifically designed for this purpose, which limits future research. To address this gap, we propose GEM-Bench, the first comprehensive benchmark for ad-injected response generation in GEM. GEM-Bench includes three curated datasets covering both chatbot and search scenarios, a metric ontology that captures multiple dimensions of user satisfaction and engagement, and several baseline solutions implemented within an extensible multi-agent framework. Our preliminary results indicate that, while simple prompt-based methods achieve reasonable engagement such as click-through rate, they often reduce user satisfaction. In contrast, approaches that insert ads based on pre-generated ad-free responses help mitigate this issue but introduce additional overhead. These findings highlight the need for future research on designing more effective and efficient solutions for generating ad-injected responses in GEM. The benchmark and all related resources are publicly available at https://gem-bench.org/.