Creating a Large Language Model of a Philosopher
This addresses the problem of evaluating AI's ability to mimic human philosophical writing, but it is incremental as it applies an existing method to a new domain.
The researchers fine-tuned GPT-3 on philosopher Daniel C. Dennett's works to generate philosophical texts, and found that experts could distinguish Dennett's answers from machine-generated ones only 51% of the time, with the model producing answers selected more frequently than Dennett's for two out of ten questions.
Can large language models be trained to produce philosophical texts that are difficult to distinguish from texts produced by human philosophers? To address this question, we fine-tuned OpenAI's GPT-3 with the works of philosopher Daniel C. Dennett as additional training data. To explore the Dennett model, we asked the real Dennett ten philosophical questions and then posed the same questions to the language model, collecting four responses for each question without cherry-picking. We recruited 425 participants to distinguish Dennett's answer from the four machine-generated answers. Experts on Dennett's work (N = 25) succeeded 51% of the time, above the chance rate of 20% but short of our hypothesized rate of 80% correct. For two of the ten questions, the language model produced at least one answer that experts selected more frequently than Dennett's own answer. Philosophy blog readers (N = 302) performed similarly to the experts, while ordinary research participants (N = 98) were near chance distinguishing GPT-3's responses from those of an "actual human philosopher".