Closed-Form Interpretation of Neural Network Classifiers with Symbolic
  Regression Gradients
Sebastian Johann Wetzel
I introduce a unified framework for interpreting neural network classifiers
tailored toward automated scientific discovery. In contrast to neural
network-based regression, for classification, it is in general impossible to
find a one-to-one mapping from the neural network to a symbolic equation even
if the neural network itself bases its classification on a quantity that can be
written as a closed-form equation. In this paper, I embed a trained neural
network into an equivalence class of classifying functions that base their
decisions on the same quantity. I interpret neural networks by finding an
intersection between this equivalence class and human-readable equations
defined by the search space of symbolic regression. The approach is not limited
to classifiers or full neural networks and can be applied to arbitrary neurons
in hidden layers or latent spaces or to simplify the process of interpreting
neural network regressors.