A New Computationally Simple Approach for Implementing Neural Networks
  with Output Hard Constraints
Andrei V. Konstantinov Lev V. Utkin
A new computationally simple method of imposing hard convex constraints on
the neural network output values is proposed. The key idea behind the method is
to map a vector of hidden parameters of the network to a point that is
guaranteed to be inside the feasible set defined by a set of constraints. The
mapping is implemented by the additional neural network layer with constraints
for output. The proposed method is simply extended to the case when constraints
are imposed not only on the output vectors, but also on joint constraints
depending on inputs. The projection approach to imposing constraints on outputs
can simply be implemented in the framework of the proposed method. It is shown
how to incorporate different types of constraints into the proposed method,
including linear and quadratic constraints, equality constraints, and dynamic
constraints, constraints in the form of boundaries. An important feature of the
method is its computational simplicity. Complexities of the forward pass of the
proposed neural network layer by linear and quadratic constraints are O(n*m)
and O(n^2*m), respectively, where n is the number of variables, m is the number
of constraints. Numerical experiments illustrate the method by solving
optimization and classification problems. The code implementing the method is
publicly available.