"A good pun is its own reword": Can Large Language Models Understand Puns?
This addresses a gap in assessing LLMs' humor comprehension for applications in creative writing and humor creation, but it is incremental as it builds on existing tasks with new evaluation methods.
The paper tackled the problem of evaluating large language models' (LLMs) understanding of puns, finding that LLMs exhibit a 'lazy pun generation' pattern and face primary challenges in pun recognition, explanation, and generation.
Puns play a vital role in academic research due to their distinct structure and clear definition, which aid in the comprehensive analysis of linguistic humor. However, the understanding of puns in large language models (LLMs) has not been thoroughly examined, limiting their use in creative writing and humor creation. In this paper, we leverage three popular tasks, i.e., pun recognition, explanation and generation to systematically evaluate the capabilities of LLMs in pun understanding. In addition to adopting the automated evaluation metrics from prior research, we introduce new evaluation methods and metrics that are better suited to the in-context learning paradigm of LLMs. These new metrics offer a more rigorous assessment of an LLM's ability to understand puns and align more closely with human cognition than previous metrics. Our findings reveal the "lazy pun generation" pattern and identify the primary challenges LLMs encounter in understanding puns.