Comparing Prior and Learned Time Representations in Transformer Models of Timeseries
This work addresses the challenge of integrating human knowledge into timeseries models for improved robustness and trustworthiness, though it is incremental in nature.
The study compared fixed versus learned time representations in Transformer models for solar energy output prediction, finding that even with well-understood periodicities, encoding prior knowledge was difficult due to unintended side-effects.
What sets timeseries analysis apart from other machine learning exercises is that time representation becomes a primary aspect of the experiment setup, as it must adequately represent the temporal relations that are relevant for the application at hand. In the work described here we study wo different variations of the Transformer architecture: one where we use the fixed time representation proposed in the literature and one where the time representation is learned from the data. Our experiments use data from predicting the energy output of solar panels, a task that exhibits known periodicities (daily and seasonal) that is straight-forward to encode in the fixed time representation. Our results indicate that even in an experiment where the phenomenon is well-understood, it is difficult to encode prior knowledge due to side-effects that are difficult to mitigate. We conclude that research work is needed to work the human into the learning loop in ways that improve the robustness and trust-worthiness of the network.