On Minimal Depth in Neural Networks
Juan L. Valerdi
A characterization of the representability of neural networks is relevant to
comprehend their success in artificial intelligence. This study investigate two
topics on ReLU neural network expressivity and their connection with a
conjecture related to the minimum depth required for representing any
continuous piecewise linear function (CPWL). The topics are the minimal depth
representation of the sum and max operations, as well as the exploration of
polytope neural networks. For the sum operation, we establish a sufficient
condition on the minimal depth of the operands to find the minimal depth of the
operation. In contrast, regarding the max operation, a comprehensive set of
examples is presented, demonstrating that no sufficient conditions, depending
solely on the depth of the operands, would imply a minimal depth for the
operation. The study also examine the minimal depth relationship between convex
CPWL functions. On polytope neural networks, we investigate several fundamental
properties, deriving results equivalent to those of ReLU networks, such as
depth inclusions and depth computation from vertices. Notably, we compute the
minimal depth of simplices, which is strictly related to the minimal depth
conjecture in ReLU networks.