UnSeenTimeQA: Time-Sensitive Question-Answering Beyond LLMs' Memorization
This addresses the need for a clean benchmark to assess temporal reasoning in LLMs, though it is incremental as it builds on existing TSQA benchmarks.
The paper tackles the problem of evaluating LLMs' temporal reasoning without data contamination by introducing UnSeenTimeQA, a synthetic benchmark, and finds that LLMs perform poorly overall, especially on long-range dependencies and parallel events.
This paper introduces UnSeenTimeQA, a novel data contamination-free time-sensitive question-answering (TSQA) benchmark. It differs from existing TSQA benchmarks by avoiding web-searchable queries grounded in the real world. We present a series of time-sensitive event scenarios based on synthetically generated facts. It requires large language models (LLMs) to engage in genuine temporal reasoning without depending on the factual knowledge acquired during the pre-training phase. Our data generation framework enables on-demand generation of new samples, mitigating the risk of data leakage. We designed three types of time-sensitive questions to test LLMs' temporal reasoning abilities over sequential and parallel event occurrences. Our evaluation of five LLMs on synthetic fact-based TSQA reveals mixed results: while they perform well on simpler subsets, their overall performance remains inferior as compared to real world fact-based TSQA. Error analysis indicates that LLMs face difficulties in reasoning over long-range event dependencies and parallel events.