Time Machine GPT
This addresses the issue of temporal misalignment in LLMs for researchers and practitioners in dynamic contexts like time-series forecasting, though it appears incremental as it builds on existing pre-training methods.
The paper tackles the problem of large language models being trained on temporally indiscriminate data, which misaligns with language evolution, by introducing Time Machine GPT (TiMaGPT), a series of point-in-time LLMs designed to be nonprognosticative to avoid future information, with models and datasets made available.
Large language models (LLMs) are often trained on extensive, temporally indiscriminate text corpora, reflecting the lack of datasets with temporal metadata. This approach is not aligned with the evolving nature of language. Conventional methods for creating temporally adapted language models often depend on further pre-training static models on time-specific data. This paper presents a new approach: a series of point-in-time LLMs called Time Machine GPT (TiMaGPT), specifically designed to be nonprognosticative. This ensures they remain uninformed about future factual information and linguistic changes. This strategy is beneficial for understanding language evolution and is of critical importance when applying models in dynamic contexts, such as time-series forecasting, where foresight of future information can prove problematic. We provide access to both the models and training datasets.