State Derivative Normalization for Continuous-Time Deep Neural Networks
This addresses a specific numerical issue in system identification for researchers and practitioners, but it is incremental as it builds on known normalization challenges.
The paper tackles the problem of improper normalization in continuous-time state-space model estimation, which leads to numerical and optimization challenges and reduced model quality, by proposing a normalization constant at the state derivative level and showing its effectiveness on a benchmark cascaded tanks system.
The importance of proper data normalization for deep neural networks is well known. However, in continuous-time state-space model estimation, it has been observed that improper normalization of either the hidden state or hidden state derivative of the model estimate, or even of the time interval can lead to numerical and optimization challenges with deep learning based methods. This results in a reduced model quality. In this contribution, we show that these three normalization tasks are inherently coupled. Due to the existence of this coupling, we propose a solution to all three normalization challenges by introducing a normalization constant at the state derivative level. We show that the appropriate choice of the normalization constant is related to the dynamics of the to-be-identified system and we derive multiple methods of obtaining an effective normalization constant. We compare and discuss all the normalization strategies on a benchmark problem based on experimental data from a cascaded tanks system and compare our results with other methods of the identification literature.