Deep Learning with Parametric Lenses
Geoffrey S. H. Cruttwell Bruno Gavranovic Neil Ghani Paul Wilson Fabio Zanasi
We propose a categorical semantics for machine learning algorithms in terms
of lenses, parametric maps, and reverse derivative categories. This foundation
provides a powerful explanatory and unifying framework: it encompasses a
variety of gradient descent algorithms such as ADAM, AdaGrad, and Nesterov
momentum, as well as a variety of loss functions such as MSE and Softmax
cross-entropy, and different architectures, shedding new light on their
similarities and differences. Furthermore, our approach to learning has
examples generalising beyond the familiar continuous domains (modelled in
categories of smooth maps) and can be realised in the discrete setting of
Boolean and polynomial circuits. We demonstrate the practical significance of
our framework with an implementation in Python.