Locations of Characters in Narratives: Andersen and Persuasion Datasets
This work addresses the challenge of assessing reading comprehension for spatial relationships in narratives, though it is incremental as it primarily introduces new datasets.
The researchers tackled the problem of evaluating AI's spatial understanding in narratives by introducing two new manually annotated datasets (Andersen and Persuasion) and testing five Large Language Models on them, with the best model achieving 61.85% accuracy on Andersen and 56.06% on Persuasion.
The ability of machines to grasp spatial understanding within narrative contexts is an intriguing aspect of reading comprehension that continues to be studied. Motivated by the goal to test the AI's competence in understanding the relationship between characters and their respective locations in narratives, we introduce two new datasets: Andersen and Persuasion. For the Andersen dataset, we selected fifteen children's stories from "Andersen's Fairy Tales" by Hans Christian Andersen and manually annotated the characters and their respective locations throughout each story. Similarly, for the Persuasion dataset, characters and their locations in the novel "Persuasion" by Jane Austen were also manually annotated. We used these datasets to prompt Large Language Models (LLMs). The prompts are created by extracting excerpts from the stories or the novel and combining them with a question asking the location of a character mentioned in that excerpt. Out of the five LLMs we tested, the best-performing one for the Andersen dataset accurately identified the location in 61.85% of the examples, while for the Persuasion dataset, the best-performing one did so in 56.06% of the cases.