Benchmarking of LLM Detection: Comparing Two Competing Approaches
This work addresses the issue of unreliable benchmarking for LLM detection tools, which is important for researchers and practitioners in AI and cybersecurity, but it appears incremental as it focuses on dataset creation and comparison rather than novel detection methods.
The paper tackles the problem of inconsistent performance evaluation in LLM text detection by creating a standardized evaluation dataset and using it to benchmark two competing detection approaches, though no concrete performance numbers are provided in the abstract.
This article gives an overview of the field of LLM text recognition. Different approaches and implemented detectors for the recognition of LLM-generated text are presented. In addition to discussing the implementations, the article focuses on benchmarking the detectors. Although there are numerous software products for the recognition of LLM-generated text, with a focus on ChatGPT-like LLMs, the quality of the recognition (recognition rate) is not clear. Furthermore, while it can be seen that scientific contributions presenting their novel approaches strive for some kind of comparison with other approaches, the construction and independence of the evaluation dataset is often not comprehensible. As a result, discrepancies in the performance evaluation of LLM detectors are often visible due to the different benchmarking datasets. This article describes the creation of an evaluation dataset and uses this dataset to investigate the different detectors. The selected detectors are benchmarked against each other.