Self Attention with Temporal Prior: Can We Learn More from Arrow of Time?
This addresses the challenge of learning temporal biases in time series data for domains like healthcare, though it is an incremental improvement on existing attention methods.
The paper tackled the problem of attention-based models requiring large data to learn short-term temporal dependencies by proposing a method that applies learnable kernels to attention matrices, achieving exceptional classification results on Electronic Health Records tasks compared to best-performing models.
Many diverse phenomena in nature often inherently encode both short- and long-term temporal dependencies, which especially result from the direction of the flow of time. In this respect, we discovered experimental evidence suggesting that interrelations of these events are higher for closer time stamps. However, to be able for attention-based models to learn these regularities in short-term dependencies, it requires large amounts of data, which are often infeasible. This is because, while they are good at learning piece-wise temporal dependencies, attention-based models lack structures that encode biases in time series. As a resolution, we propose a simple and efficient method that enables attention layers to better encode the short-term temporal bias of these data sets by applying learnable, adaptive kernels directly to the attention matrices. We chose various prediction tasks for the experiments using Electronic Health Records (EHR) data sets since they are great examples with underlying long- and short-term temporal dependencies. Our experiments show exceptional classification results compared to best-performing models on most tasks and data sets.