Evaluating Large Language Model Creativity from a Literary Perspective
This work addresses the challenge of evaluating LLM creativity for writers and researchers in computational creativity, but it is incremental as it builds on existing prompting methods with a qualitative focus.
This paper tackled the problem of assessing large language models' potential as assistive tools in creative writing by conducting an in-depth case study, finding that the sophistication of the results depends on the sophistication of the prompting strategies used.
This paper assesses the potential for large language models (LLMs) to serve as assistive tools in the creative writing process, by means of a single, in-depth case study. In the course of the study, we develop interactive and multi-voice prompting strategies that interleave background descriptions (scene setting, plot elements), instructions that guide composition, samples of text in the target style, and critical discussion of the given samples. We qualitatively evaluate the results from a literary critical perspective, as well as from the standpoint of computational creativity (a sub-field of artificial intelligence). Our findings lend support to the view that the sophistication of the results that can be achieved with an LLM mirrors the sophistication of the prompting.