The BS-meter: A ChatGPT-Trained Instrument to Detect Sloppy Language-Games
This addresses concerns about sloppy language in AI-generated text, offering a tool to detect such patterns, though it is incremental in applying existing methods to new contexts.
The paper analyzed language generated by ChatGPT compared to scientific publications, showing that a statistical model can reliably link ChatGPT's 'artificial bullshit' to political and workplace bullshit in human language.
What can we learn about language from studying how it is used by ChatGPT and other large language model (LLM)-based chatbots? In this paper, we analyse the distinctive character of language generated by ChatGPT, in relation to questions raised by natural language processing pioneer, and student of Wittgenstein, Margaret Masterman. Following frequent complaints that LLM-based chatbots produce "slop," or even "bullshit," in the sense of Frankfurt's popular monograph On Bullshit, we conduct an empirical study to contrast the language of 1,000 scientific publications with typical text generated by ChatGPT. We then explore whether the same language features can be detected in two well-known contexts of social dysfunction: George Orwell's critique of political speech, and David Graeber's characterisation of bullshit jobs. Using simple hypothesis-testing methods, we demonstrate that a statistical model of sloppy bullshit can reliably relate the Frankfurtian artificial bullshit of ChatGPT to the political and workplace functions of bullshit as observed in natural human language.