Riddle Quest : The Enigma of Words
This provides a lightweight tool for examining reasoning coverage and ambiguity handling in language models, which is incremental as it applies existing methods to a new domain.
The paper tackled the problem of evaluating how well large language models handle ambiguity in reasoning by creating and testing analogy-based riddles, finding that models often miss valid interpretations beyond the main intended answer.
Riddles are concise linguistic puzzles that describe an object or idea through indirect, figurative, or playful clues. They are a longstanding form of creative expression, requiring the solver to interpret hints, recognize patterns, and draw inferences to identify the answers. In this work, we introduce a simple pipeline for creating and evaluating analogy-based riddles. The system includes a triples creator that builds structured facts about a concept, a semantic mapper that selects attributes useful for analogy, a stylized generator that turns them into riddle clues, and a validator that collects all possible answers the riddle could point to. We use this validator to study whether large language models can recover the full answer set for different riddle types. Our case study shows that while models often guess the main intended answer, they frequently miss other valid interpretations. This highlights the value of riddles as a lightweight tool for examining reasoning coverage and ambiguity handling in language models.