Time-Stamped Language Model: Teaching Language Models to Understand the Flow of Events
This work addresses procedural text understanding for NLP applications, representing an incremental advance with specific gains.
The paper tackled the challenge of tracking entities in procedural texts by formulating it as a question answering problem and introducing a Time-Stamped Language Model (TSLM) to encode event flow, resulting in a 3.1% F1 score improvement on the Propara dataset and better performance on the NPN-Cooking dataset.
Tracking entities throughout a procedure described in a text is challenging due to the dynamic nature of the world described in the process. Firstly, we propose to formulate this task as a question answering problem. This enables us to use pre-trained transformer-based language models on other QA benchmarks by adapting those to the procedural text understanding. Secondly, since the transformer-based language models cannot encode the flow of events by themselves, we propose a Time-Stamped Language Model~(TSLM model) to encode event information in LMs architecture by introducing the timestamp encoding. Our model evaluated on the Propara dataset shows improvements on the published state-of-the-art results with a $3.1\%$ increase in F1 score. Moreover, our model yields better results on the location prediction task on the NPN-Cooking dataset. This result indicates that our approach is effective for procedural text understanding in general.