Gradient Networks
Shreyas Chaudhari Srinivasa Pranav José M. F. Moura
Directly parameterizing and learning gradients of functions has widespread
significance, with specific applications in optimization, generative modeling,
and optimal transport. This paper introduces gradient networks (GradNets):
novel neural network architectures that parameterize gradients of various
function classes. GradNets exhibit specialized architectural constraints that
ensure correspondence to gradient functions. We provide a comprehensive GradNet
design framework that includes methods for transforming GradNets into monotone
gradient networks (mGradNets), which are guaranteed to represent gradients of
convex functions. We establish the approximation capabilities of the proposed
GradNet and mGradNet. Our results demonstrate that these networks universally
approximate the gradients of (convex) functions. Furthermore, these networks
can be customized to correspond to specific spaces of (monotone) gradient
functions, including gradients of transformed sums of (convex) ridge functions.
Our analysis leads to two distinct GradNet architectures, GradNet-C and
GradNet-M, and we describe the corresponding monotone versions, mGradNet-C and
mGradNet-M. Our empirical results show that these architectures offer efficient
parameterizations and outperform popular methods in gradient field learning
tasks.