Drafting Event Schemas using Language Models
This work addresses the need for easier distillation of event knowledge into schemas, which can aid in explainable predictions and forecasting, but it is incremental as it builds on existing methods for schema creation.
The paper tackles the problem of creating event schemas for complex events by using large language models to draft them in natural language, achieving moderate recall against datasets and improved results with multiple prompts and samples.
Past work has studied event prediction and event language modeling, sometimes mediated through structured representations of knowledge in the form of event schemas. Such schemas can lead to explainable predictions and forecasting of unseen events given incomplete information. In this work, we look at the process of creating such schemas to describe complex events. We use large language models (LLMs) to draft schemas directly in natural language, which can be further refined by human curators as necessary. Our focus is on whether we can achieve sufficient diversity and recall of key events and whether we can produce the schemas in a sufficiently descriptive style. We show that large language models are able to achieve moderate recall against schemas taken from two different datasets, with even better results when multiple prompts and multiple samples are combined. Moreover, we show that textual entailment methods can be used for both matching schemas to instances of events as well as evaluating overlap between gold and predicted schemas. Our method paves the way for easier distillation of event knowledge from large language model into schemas.