Beemo: Benchmark of Expert-edited Machine-generated Outputs
This addresses the need for better benchmarks in MGT detection for researchers and practitioners, though it is incremental as it extends existing benchmark designs.
The paper tackles the problem that existing benchmarks for machine-generated texts (MGTs) fail to capture practical multi-author scenarios where users edit LLM outputs, by introducing Beemo, a benchmark with 6.5k expert-edited texts and 13.1k LLM-edited texts, and finds that expert-based editing evades MGT detection while LLM-edited texts are not recognized as human-written.
The rapid proliferation of large language models (LLMs) has increased the volume of machine-generated texts (MGTs) and blurred text authorship in various domains. However, most existing MGT benchmarks include single-author texts (human-written and machine-generated). This conventional design fails to capture more practical multi-author scenarios, where the user refines the LLM response for natural flow, coherence, and factual correctness. Our paper introduces the Benchmark of Expert-edited Machine-generated Outputs (Beemo), which includes 6.5k texts written by humans, generated by ten instruction-finetuned LLMs, and edited by experts for various use cases, ranging from creative writing to summarization. Beemo additionally comprises 13.1k machine-generated and LLM-edited texts, allowing for diverse MGT detection evaluation across various edit types. We document Beemo's creation protocol and present the results of benchmarking 33 configurations of MGT detectors in different experimental setups. We find that expert-based editing evades MGT detection, while LLM-edited texts are unlikely to be recognized as human-written. Beemo and all materials are publicly available.