Creativity Benchmark: A benchmark for marketing creativity for large language models
This addresses the need for expert human evaluation in assessing LLM creativity for marketing, but it is incremental as it focuses on benchmarking rather than new methods.
The authors tackled the problem of evaluating large language models (LLMs) in marketing creativity by introducing the Creativity Benchmark, which uses human preferences from 678 creatives over 11,012 comparisons, showing tightly clustered model performance with a head-to-head win probability of 0.61 and weak correlations for automated judges.
We introduce Creativity Benchmark, an evaluation framework for large language models (LLMs) in marketing creativity. The benchmark covers 100 brands (12 categories) and three prompt types (Insights, Ideas, Wild Ideas). Human pairwise preferences from 678 practising creatives over 11,012 anonymised comparisons, analysed with Bradley-Terry models, show tightly clustered performance with no model dominating across brands or prompt types: the top-bottom spread is $Δθ\approx 0.45$, which implies a head-to-head win probability of $0.61$; the highest-rated model beats the lowest only about $61\%$ of the time. We also analyse model diversity using cosine distances to capture intra- and inter-model variation and sensitivity to prompt reframing. Comparing three LLM-as-judge setups with human rankings reveals weak, inconsistent correlations and judge-specific biases, underscoring that automated judges cannot substitute for human evaluation. Conventional creativity tests also transfer only partially to brand-constrained tasks. Overall, the results highlight the need for expert human evaluation and diversity-aware workflows.