The Real Tropical Geometry of Neural Networks
Marie-Charlotte Brandenburg Georg Loho Guido Montúfar
We consider a binary classifier defined as the sign of a tropical rational
function, that is, as the difference of two convex piecewise linear functions.
The parameter space of ReLU neural networks is contained as a semialgebraic set
inside the parameter space of tropical rational functions. We initiate the
study of two different subdivisions of this parameter space: a subdivision into
semialgebraic sets, on which the combinatorial type of the decision boundary is
fixed, and a subdivision into a polyhedral fan, capturing the combinatorics of
the partitions of the dataset. The sublevel sets of the 0/1-loss function arise
as subfans of this classification fan, and we show that the level-sets are not
necessarily connected. We describe the classification fan i) geometrically, as
normal fan of the activation polytope, and ii) combinatorially through a list
of properties of associated bipartite graphs, in analogy to covector axioms of
oriented matroids and tropical oriented matroids. Our findings extend and
refine the connection between neural networks and tropical geometry by
observing structures established in real tropical geometry, such as positive
tropicalizations of hypersurfaces and tropical semialgebraic sets.