HT-Transformer: Event Sequences Classification by Accumulating Prefix Information with History Tokens
This addresses a key limitation in transformer models for sequential data classification, offering a novel method to enhance performance in domains like finance and healthcare, though it is incremental as it builds on existing transformer frameworks.
The paper tackles the performance gap of transformers compared to RNNs in event sequence classification by identifying the lack of a compact state representation and poor local context capture, and introduces history tokens to accumulate historical information during pretraining, achieving significant improvements in finance, e-commerce, and healthcare tasks.
Deep learning has achieved remarkable success in modeling sequential data, including event sequences, temporal point processes, and irregular time series. Recently, transformers have largely replaced recurrent networks in these tasks. However, transformers often underperform RNNs in classification tasks where the objective is to predict future targets. The reason behind this performance gap remains largely unexplored. In this paper, we identify a key limitation of transformers: the absence of a single state vector that provides a compact and effective representation of the entire sequence. Additionally, we show that contrastive pretraining of embedding vectors fails to capture local context, which is crucial for accurate prediction. To address these challenges, we introduce history tokens, a novel concept that facilitates the accumulation of historical information during next-token prediction pretraining. Our approach significantly improves transformer-based models, achieving impressive results in finance, e-commerce, and healthcare tasks. The code is publicly available on GitHub.