Extraction of nonlinearity in neural networks and model compression with
  Koopman operator
Naoki Sugishita Kayo Kinjo Jun Ohkubo
Nonlinearity plays a crucial role in deep neural networks. In this paper, we
first investigate the degree to which the nonlinearity of the neural network is
essential. For this purpose, we employ the Koopman operator, extended dynamic
mode decomposition, and the tensor-train format. The results imply that
restricted nonlinearity is enough for the classification of handwritten
numbers. Then, we propose a model compression method for deep neural networks,
which could be beneficial to handling large networks in resource-constrained
environments. Leveraging the Koopman operator, the proposed method enables us
to use linear algebra in the internal processing of neural networks. We
numerically show that the proposed method performs comparably or better than
conventional methods in highly compressed model settings for the handwritten
number recognition task.