Semi-Structured Object Sequence Encoders
This work addresses modeling challenges for semi-structured data like website logs, offering incremental improvements in handling longer sequences for specific domains.
The paper tackles the problem of modeling semi-structured object sequences, such as user activity or machine logs, by proposing a two-part approach that encodes key-value pairs over time and uses self-attention, resulting in improved performance on multiple real-world prediction tasks compared to existing methods.
In this paper we explore the task of modeling semi-structured object sequences; in particular, we focus our attention on the problem of developing a structure-aware input representation for such sequences. Examples of such data include user activity on websites, machine logs, and many others. This type of data is often represented as a sequence of sets of key-value pairs over time and can present modeling challenges due to an ever-increasing sequence length. We propose a two-part approach, which first considers each key independently and encodes a representation of its values over time; we then self-attend over these value-aware key representations to accomplish a downstream task. This allows us to operate on longer object sequences than existing methods. We introduce a novel shared-attention-head architecture between the two modules and present an innovative training schedule that interleaves the training of both modules with shared weights for some attention heads. Our experiments on multiple prediction tasks using real-world data demonstrate that our approach outperforms a unified network with hierarchical encoding, as well as other methods including a record-centric representation and a flattened representation of the sequence.