FACTTRACK: Time-Aware World State Tracking in Story Outlines
This addresses the challenge of factual accuracy in language models for applications like story generation, though it is incremental as it builds on existing contradiction detection methods.
The paper tackles the problem of detecting and correcting factual contradictions in language model outputs by proposing FACTTRACK, a method that tracks atomic facts with time-aware validity intervals, and finds that it outperforms baselines using LLaMA2-7B-Chat and GPT4, achieving comparable or superior performance.
While accurately detecting and correcting factual contradictions in language model outputs has become increasingly important as their capabilities improve, doing so is highly challenging. We propose a novel method, FACTTRACK, for tracking atomic facts and addressing factual contradictions. Crucially, FACTTRACK also maintains time-aware validity intervals for each fact, allowing for change over time. At a high level, FACTTRACK consists of a four-step pipeline to update a world state data structure for each new event: (1) decompose the event into directional atomic facts; (2) determine the validity interval of each atomic fact using the world state; (3) detect contradictions with existing facts in the world state; and finally (4) add new facts to the world state and update existing atomic facts. When we apply FACTTRACK to contradiction detection on structured story outlines, we find that FACTTRACK using LLaMA2-7B-Chat substantially outperforms a fair baseline using LLaMA2-7B-Chat, and achieves performance comparable to a GPT4 baseline. Moreover, when using GPT4, FACTTRACK significantly outperforms the GPT4 baseline.