Are LLMs Good Cryptic Crossword Solvers?
This addresses the challenge of evaluating LLMs on complex language tasks like cryptic crosswords, but it is incremental as it tests existing models on a new benchmark.
The paper tackled the problem of solving cryptic crosswords using large language models (LLMs), establishing benchmark results for LLaMA2, Mistral, and ChatGPT, and found that their performance is far from human levels.
Cryptic crosswords are puzzles that rely not only on general knowledge but also on the solver's ability to manipulate language on different levels and deal with various types of wordplay. Previous research suggests that solving such puzzles is a challenge even for modern NLP models. However, the abilities of large language models (LLMs) have not yet been tested on this task. In this paper, we establish the benchmark results for three popular LLMs -- LLaMA2, Mistral, and ChatGPT -- showing that their performance on this task is still far from that of humans.