Statistical Mechanics of Dynamical System Identification
Andrei A. Klishin Joseph Bakarji J. Nathan Kutz Krithika Manohar
Recovering dynamical equations from observed noisy data is the central
challenge of system identification. We develop a statistical mechanical
approach to analyze sparse equation discovery algorithms, which typically
balance data fit and parsimony through a trial-and-error selection of
hyperparameters. In this framework, statistical mechanics offers tools to
analyze the interplay between complexity and fitness, in analogy to that done
between entropy and energy. To establish this analogy, we define the
optimization procedure as a two-level Bayesian inference problem that separates
variable selection from coefficient values and enables the computation of the
posterior parameter distribution in closed form. A key advantage of employing
statistical mechanical concepts, such as free energy and the partition
function, is in the quantification of uncertainty, especially in in the
low-data limit; frequently encountered in real-world applications. As the data
volume increases, our approach mirrors the thermodynamic limit, leading to
distinct sparsity- and noise-induced phase transitions that delineate correct
from incorrect identification. This perspective of sparse equation discovery,
is versatile and can be adapted to various other equation discovery algorithms.