State Derivative Normalization for Continuous-Time Deep Neural Networks
Jonas Weigand Gerben I. Beintema Jonas Ulmen Daniel Görges Roland Tóth Maarten Schoukens Martin Ruskowski
The importance of proper data normalization for deep neural networks is well
known. However, in continuous-time state-space model estimation, it has been
observed that improper normalization of either the hidden state or hidden state
derivative of the model estimate, or even of the time interval can lead to
numerical and optimization challenges with deep learning based methods. This
results in a reduced model quality. In this contribution, we show that these
three normalization tasks are inherently coupled. Due to the existence of this
coupling, we propose a solution to all three normalization challenges by
introducing a normalization constant at the state derivative level. We show
that the appropriate choice of the normalization constant is related to the
dynamics of the to-be-identified system and we derive multiple methods of
obtaining an effective normalization constant. We compare and discuss all the
normalization strategies on a benchmark problem based on experimental data from
a cascaded tanks system and compare our results with other methods of the
identification literature.