Globally injective and bijective neural operators
Takashi Furuya Michael Puthawala Matti Lassas Maarten V. de Hoop
Recently there has been great interest in operator learning, where networks
learn operators between function spaces from an essentially
infinite-dimensional perspective. In this work we present results for when the
operators learned by these networks are injective and surjective. As a warmup,
we combine prior work in both the finite-dimensional ReLU and operator learning
setting by giving sharp conditions under which ReLU layers with linear neural
operators are injective. We then consider the case the case when the activation
function is pointwise bijective and obtain sufficient conditions for the layer
to be injective. We remark that this question, while trivial in the finite-rank
case, is subtler in the infinite-rank case and is proved using tools from
Fredholm theory. Next, we prove that our supplied injective neural operators
are universal approximators and that their implementation, with finite-rank
neural networks, are still injective. This ensures that injectivity is not
`lost' in the transcription from analytical operators to their finite-rank
implementation with networks. Finally, we conclude with an increase in
abstraction and consider general conditions when subnetworks, which may be many
layers deep, are injective and surjective and provide an exact inversion from a
`linearization.' This section uses general arguments from Fredholm theory and
Leray-Schauder degree theory for non-linear integral equations to analyze the
mapping properties of neural operators in function spaces. These results apply
to subnetworks formed from the layers considered in this work, under natural
conditions. We believe that our work has applications in Bayesian UQ where
injectivity enables likelihood estimation and in inverse problems where
surjectivity and injectivity corresponds to existence and uniqueness,
respectively.