A provable control of sensitivity of neural networks through a direct
  parameterization of the overall bi-Lipschitzness
Yuri Kinoshita Taro Toyoizumi
While neural networks can enjoy an outstanding flexibility and exhibit
unprecedented performance, the mechanism behind their behavior is still not
well-understood. To tackle this fundamental challenge, researchers have tried
to restrict and manipulate some of their properties in order to gain new
insights and better control on them. Especially, throughout the past few years,
the concept of \emph{bi-Lipschitzness} has been proved as a beneficial
inductive bias in many areas. However, due to its complexity, the design and
control of bi-Lipschitz architectures are falling behind, and a model that is
precisely designed for bi-Lipschitzness realizing a direct and simple control
of the constants along with solid theoretical analysis is lacking. In this
work, we investigate and propose a novel framework for bi-Lipschitzness that
can achieve such a clear and tight control based on convex neural networks and
the Legendre-Fenchel duality. Its desirable properties are illustrated with
concrete experiments. We also apply this framework to uncertainty estimation
and monotone problem settings to illustrate its broad range of applications.