Long short-term relevance learning
This work addresses the challenge of overfitting and architectural design in LSTM networks for time-dependent applications like structural analysis, but it is incremental as it builds on existing LSTM methods with a Bayesian adaptation.
The authors tackled the problem of incorporating prior knowledge and measurement uncertainties into LSTM neural networks by introducing a sparse Bayesian training algorithm, resulting in a scheme that is less prone to overfitting and can approximate time-dependent solutions with smaller datasets, as demonstrated on a structural nonlinear finite element application with satisfying accuracy at reasonable cost.
To incorporate prior knowledge as well as measurement uncertainties in the traditional long short term memory (LSTM) neural networks, an efficient sparse Bayesian training algorithm is introduced to the network architecture. The proposed scheme automatically determines relevant neural connections and adapts accordingly, in contrast to the classical LSTM solution. Due to its flexibility, the new LSTM scheme is less prone to overfitting, and hence can approximate time dependent solutions by use of a smaller data set. On a structural nonlinear finite element application we show that the self-regulating framework does not require prior knowledge of a suitable network architecture and size, while ensuring satisfying accuracy at reasonable computational cost.