Motif distribution and function of sparse deep neural networks
Olivia T. Zahn Thomas L. Daniel J. Nathan Kutz
We characterize the connectivity structure of feed-forward, deep neural
networks (DNNs) using network motif theory. To address whether a particular
motif distribution is characteristic of the training task, or function of the
DNN, we compare the connectivity structure of 350 DNNs trained to simulate a
bio-mechanical flight control system with different randomly initialized
parameters. We develop and implement algorithms for counting second- and
third-order motifs and calculate their significance using their Z-score. The
DNNs are trained to solve the inverse problem of the flight dynamics model in
Bustamante, et al. (2022) (i.e., predict the controls necessary for controlled
flight from the initial and final state-space inputs) and are sparsified
through an iterative pruning and retraining algorithm Zahn, et al. (2022). We
show that, despite random initialization of network parameters, enforced
sparsity causes DNNs to converge to similar connectivity patterns as
characterized by their motif distributions. The results suggest how neural
network function can be encoded in motif distributions, suggesting a variety of
experiments for informing function and control.