A Highly Clean Recipe Dataset with Ingredient States Annotation for State Probing Task
This work addresses a domain-specific challenge in AI for cooking recipe understanding, but it is incremental as it applies existing state probing methods to a new dataset and task.
The authors tackled the problem of large language models (LLMs) struggling to track intermediate ingredient states in cooking recipes due to omitted details, by creating a new Japanese recipe dataset with annotated state changes and designing tasks to evaluate LLM understanding. Their experiments showed that learning ingredient state knowledge improved LLM performance on cooking processes, achieving results comparable to commercial models.
Large Language Models (LLMs) are trained on a vast amount of procedural texts, but they do not directly observe real-world phenomena. In the context of cooking recipes, this poses a challenge, as intermediate states of ingredients are often omitted, making it difficult for models to track ingredient states and understand recipes accurately. In this paper, we apply state probing, a method for evaluating a language model's understanding of the world, to the domain of cooking. We propose a new task and dataset for evaluating how well LLMs can recognize intermediate ingredient states during cooking procedures. We first construct a new Japanese recipe dataset with clear and accurate annotations of ingredient state changes, collected from well-structured and controlled recipe texts. Using this dataset, we design three novel tasks to evaluate whether LLMs can track ingredient state transitions and identify ingredients present at intermediate steps. Our experiments with widely used LLMs, such as Llama3.1-70B and Qwen2.5-72B, show that learning ingredient state knowledge improves their understanding of cooking processes, achieving performance comparable to commercial LLMs. The dataset are publicly available at: https://huggingface.co/datasets/mashi6n/nhkrecipe-100-anno-1