SOHET: Sequence Of Heterogeneous Events Transformer with Self-Supervised Pre-Training
Provides a new architecture and self-supervised pre-training for heterogeneous event sequences, improving fraud detection for e-commerce platforms.
SOHET introduces a hierarchical transformer for heterogeneous event streams, achieving a 5.8% improvement over baselines on a Booking.com fraud detection task and matching/exceeding state-of-the-art on 6/8 EBES benchmarks.
Many machine learning applications rely on heterogeneous event streams to make predictions, either causally as events arrive or bidirectionally over complete sequences. We propose SOHET (Sequence Of Heterogeneous Events Transformer), a hierarchical architecture combining event-type-specific tabular encoders with temporal and type embeddings, processed by a causal or bidirectional transformer. We introduce three self-supervised pre-training objectives for the causal setting. On a proprietary large-scale real-world Booking.com fraud detection task with 17 event types, SOHET outperforms FlexTPP, NAPPT, and CIPPT by 5.8%. Pre-training yields an additional 2.6% gain and 2.4% faster convergence. On the EBES benchmark, bidirectional SOHET matches or exceeds the published best on 6 out of 8 tasks.