Towards a Robust Detection of Language Model Generated Text: Is ChatGPT that Easy to Detect?
This addresses the challenge of robust AI-generated text detection for French language applications, but it is incremental as it adapts existing methods to a new language and dataset.
The paper tackles the problem of detecting ChatGPT-generated text in French by translating an English dataset and training classifiers, finding that detectors are effective in in-domain settings but vulnerable to out-of-domain data and attacks.
Recent advances in natural language processing (NLP) have led to the development of large language models (LLMs) such as ChatGPT. This paper proposes a methodology for developing and evaluating ChatGPT detectors for French text, with a focus on investigating their robustness on out-of-domain data and against common attack schemes. The proposed method involves translating an English dataset into French and training a classifier on the translated data. Results show that the detectors can effectively detect ChatGPT-generated text, with a degree of robustness against basic attack techniques in in-domain settings. However, vulnerabilities are evident in out-of-domain contexts, highlighting the challenge of detecting adversarial text. The study emphasizes caution when applying in-domain testing results to a wider variety of content. We provide our translated datasets and models as open-source resources. https://gitlab.inria.fr/wantoun/robust-chatgpt-detection