Autoencoders for a manifold learning problem with a Jacobian rank
  constraint
Rustem Takhanov Y. Sultan Abylkairov Maxat Tezekbayev
We formulate the manifold learning problem as the problem of finding an
operator that maps any point to a close neighbor that lies on a ``hidden''
$k$-dimensional manifold. We call this operator the correcting function. Under
this formulation, autoencoders can be viewed as a tool to approximate the
correcting function. Given an autoencoder whose Jacobian has rank $k$, we
deduce from the classical Constant Rank Theorem that its range has a structure
of a $k$-dimensional manifold.
  A $k$-dimensionality of the range can be forced by the architecture of an
autoencoder (by fixing the dimension of the code space), or alternatively, by
an additional constraint that the rank of the autoencoder mapping is not
greater than $k$. This constraint is included in the objective function as a
new term, namely a squared Ky-Fan $k$-antinorm of the Jacobian function. We
claim that this constraint is a factor that effectively reduces the dimension
of the range of an autoencoder, additionally to the reduction defined by the
architecture. We also add a new curvature term into the objective. To conclude,
we experimentally compare our approach with the CAE+H method on synthetic and
real-world datasets.