AdTEC: A Unified Benchmark for Evaluating Text Quality in Search Engine Advertising
This addresses the need for standardized evaluation of automatically generated ad texts in advertising operations, though it is incremental as it builds on existing benchmarks and methods.
The paper tackles the problem of evaluating text quality in search engine advertising by introducing AdTEC, the first public benchmark for assessing ad texts from multiple perspectives, showing that pre-trained language models reach practical levels in some tasks but humans still outperform in certain domains.
With the increase in the fluency of ad texts automatically created by natural language generation technology, there is high demand to verify the quality of these creatives in a real-world setting. We propose AdTEC (Ad Text Evaluation Benchmark by CyberAgent), the first public benchmark to evaluate ad texts from multiple perspectives within practical advertising operations. Our contributions are as follows: (i) Defining five tasks for evaluating the quality of ad texts, as well as building a Japanese dataset based on the practical operational experiences of building a Japanese dataset based on the practical operational experiences of advertising agencies, which are typically kept in-house. (ii) Validating the performance of existing pre-trained language models (PLMs) and human evaluators on the dataset. (iii) Analyzing the characteristics and providing challenges of the benchmark. The results show that while PLMs have already reached practical usage level in several tasks, humans still outperform in certain domains, implying that there is significant room for improvement in this area.