Temporal Tokenization Strategies for Event Sequence Modeling with Large Language Models
This addresses the under-explored challenge of temporal representation for event sequence modeling, which is incremental as it compares existing strategies rather than introducing a new method.
This paper tackles the problem of representing continuous time in temporal event sequence modeling with large language models by empirically comparing five temporal tokenization strategies on real-world datasets. The result shows that no single strategy is universally superior, with performance depending on aligning the tokenizer with the data's statistical properties, such as log-based strategies for skewed distributions and human-centric formats for mixed modalities.
Representing continuous time is a critical and under-explored challenge in modeling temporal event sequences with large language models (LLMs). Various strategies like byte-level representations or calendar tokens have been proposed. However, the optimal approach remains unclear, especially given the diverse statistical distributions of real-world event data, which range from smooth log-normal to discrete, spiky patterns. This paper presents the first empirical study of temporal tokenization for event sequences, comparing distinct encoding strategies: naive numeric strings, high-precision byte-level representations, human-semantic calendar tokens, classic uniform binning, and adaptive residual scalar quantization. We evaluate these strategies by fine-tuning LLMs on real-world datasets that exemplify these diverse distributions. Our analysis reveals that no single strategy is universally superior; instead, prediction performance depends heavily on aligning the tokenizer with the data's statistical properties, with log-based strategies excelling on skewed distributions and human-centric formats proving robust for mixed modalities.