Linearization of ReLU Activation Function for Neural Network-Embedded
  Optimization:Optimal Day-Ahead Energy Scheduling
Cunzhi Zhao Xingpeng Li
Neural networks have been widely applied in the power system area. They can
be used for better predicting input information and modeling system performance
with increased accuracy. In some applications such as battery degradation
neural network-based microgrid day-ahead energy scheduling, the input features
of the trained learning model are variables to be solved in optimization models
that enforce limits on the output of the same learning model. This will create
a neural network-embedded optimization problem; the use of nonlinear activation
functions in the neural network will make such problems extremely hard to solve
if not unsolvable. To address this emerging challenge, this paper investigated
different methods for linearizing the nonlinear activation functions with a
particular focus on the widely used rectified linear unit (ReLU) function. Four
linearization methods tailored for the ReLU activation function are developed,
analyzed and compared in this paper. Each method employs a set of linear
constraints to replace the ReLU function, effectively linearizing the
optimization problem, which can overcome the computational challenges
associated with the nonlinearity of the neural network model. These proposed
linearization methods provide valuable tools for effectively solving
optimization problems that integrate neural network models with ReLU activation
functions.