Understanding deep neural networks through the lens of their
  non-linearity
Quentin Bouniot Ievgen Redko Anton Mallasto Charlotte Laclau Karol Arndt Oliver Struckmeier Markus Heinonen Ville Kyrki Samuel Kaski
The remarkable success of deep neural networks (DNN) is often attributed to
their high expressive power and their ability to approximate functions of
arbitrary complexity. Indeed, DNNs are highly non-linear models, and activation
functions introduced into them are largely responsible for this. While many
works studied the expressive power of DNNs through the lens of their
approximation capabilities, quantifying the non-linearity of DNNs or of
individual activation functions remains an open problem. In this paper, we
propose the first theoretically sound solution to track non-linearity
propagation in deep neural networks with a specific focus on computer vision
applications. Our proposed affinity score allows us to gain insights into the
inner workings of a wide range of different architectures and learning
paradigms. We provide extensive experimental results that highlight the
practical utility of the proposed affinity score and its potential for
long-reaching applications.