Temporal Weights
This addresses the limitation of static weight representations in neural networks for time series modeling, offering incremental improvements in efficiency and performance.
The paper tackled the problem of static weights in neural networks by introducing Temporal Weights (TW) with interacting dynamics, resulting in better performance, smaller models, and improved data efficiency on sparse, irregularly sampled time series datasets.
In artificial neural networks, weights are a static representation of synapses. However, synapses are not static, they have their own interacting dynamics over time. To instill weights with interacting dynamics, we use a model describing synchronization that is capable of capturing core mechanisms of a range of neural and general biological phenomena over time. An ideal fit for these Temporal Weights (TW) are Neural ODEs, with continuous dynamics and a dependency on time. The resulting recurrent neural networks efficiently model temporal dynamics by computing on the ordering of sequences, and the length and scale of time. By adding temporal weights to a model, we demonstrate better performance, smaller models, and data efficiency on sparse, irregularly sampled time series datasets.