Temporal Alignment of LLMs through Cycle Encoding for Long-Range Time Representations
This addresses temporal misalignment issues in LLMs for applications requiring long-range time understanding, but it is incremental as it builds on existing encoding techniques.
The paper tackles the problem of temporal misalignment in large language models (LLMs) over long time spans by proposing a method called 'Ticktack' that uses sexagenary year expressions and polar coordinates for encoding, resulting in improved performance on time-related tasks as validated by a new benchmark.
Large language models (LLMs) suffer from temporal misalignment issues especially across long span of time. The issue arises from knowing that LLMs are trained on large amounts of data where temporal information is rather sparse over long times, such as thousands of years, resulting in insufficient learning or catastrophic forgetting by the LLMs. This paper proposes a methodology named "Ticktack" for addressing the LLM's long-time span misalignment in a yearly setting. Specifically, we first propose to utilize the sexagenary year expression instead of the Gregorian year expression employed by LLMs, achieving a more uniform distribution in yearly granularity. Then, we employ polar coordinates to model the sexagenary cycle of 60 terms and the year order within each term, with additional temporal encoding to ensure LLMs understand them. Finally, we present a temporal representational alignment approach for post-training LLMs that effectively distinguishes time points with relevant knowledge, hence improving performance on time-related tasks, particularly over a long period. We also create a long time span benchmark for evaluation. Experimental results prove the effectiveness of our proposal.