DyWPE: Signal-Aware Dynamic Wavelet Positional Encoding for Time Series Transformers
This addresses a limitation in time series transformers for applications like biomedical analysis, though it appears incremental as it builds on existing positional encoding methods.
The paper tackles the problem of signal-agnostic positional encoding in transformers for time series analysis by introducing DyWPE, a signal-aware framework using wavelet transforms. It demonstrates consistent improvements over eight existing methods, achieving an average 9.1% relative gain in biomedical signals.
Existing positional encoding methods in transformers are fundamentally signal-agnostic, deriving positional information solely from sequence indices while ignoring the underlying signal characteristics. This limitation is particularly problematic for time series analysis, where signals exhibit complex, non-stationary dynamics across multiple temporal scales. We introduce Dynamic Wavelet Positional Encoding (DyWPE), a novel signal-aware framework that generates positional embeddings directly from input time series using the Discrete Wavelet Transform (DWT). Comprehensive experiments in ten diverse time series datasets demonstrate that DyWPE consistently outperforms eight existing state-of-the-art positional encoding methods, achieving average relative improvements of 9.1\% compared to baseline sinusoidal absolute position encoding in biomedical signals, while maintaining competitive computational efficiency.