Looking around you: external information enhances representations for event sequences
This addresses the issue of isolated sequence processing for users in dynamic domains, but it is incremental as it builds on existing encoders.
The paper tackled the problem of event sequence models ignoring contextual information from co-occurring sequences, which limits performance in fast-evolving domains like finance and e-commerce. The result was that their method, using Kernel attention aggregation, improved ROC-AUC scores across six diverse datasets, with mean pooling also providing significant gains.
Representation learning produces models in different domains, such as store purchases, client transactions, and general people's behaviour. However, such models for event sequences usually process each sequence in isolation, ignoring context from ones that co-occur in time. This limitation is particularly problematic in domains with fast-evolving conditions, like finance and e-commerce, or when certain sequences lack recent events. We develop a method that aggregates information from multiple user representations, augmenting a specific user for a scenario of multiple co-occurring event sequences, achieving better quality than processing each sequence independently. Our study considers diverse aggregation approaches, ranging from simple pooling techniques to trainable attention-based Kernel attention aggregation, that can highlight more complex information flow from other users. The proposed methods operate on top of an existing encoder and support its efficient fine-tuning. Across six diverse event sequence datasets (finance, e-commerce, education, etc.) and downstream tasks, Kernel attention improves ROC-AUC scores, both with and without fine-tuning, while mean pooling yields a smaller but still significant gain.