A Momentum Accelerated Algorithm for ReLU-based Nonlinear Matrix
  Decomposition
Qingsong Wang Chunfeng Cui Deren Han
Recently, there has been a growing interest in the exploration of Nonlinear
Matrix Decomposition (NMD) due to its close ties with neural networks. NMD aims
to find a low-rank matrix from a sparse nonnegative matrix with a per-element
nonlinear function. A typical choice is the Rectified Linear Unit (ReLU)
activation function. To address over-fitting in the existing ReLU-based NMD
model (ReLU-NMD), we propose a Tikhonov regularized ReLU-NMD model, referred to
as ReLU-NMD-T. Subsequently, we introduce a momentum accelerated algorithm for
handling the ReLU-NMD-T model. A distinctive feature, setting our work apart
from most existing studies, is the incorporation of both positive and negative
momentum parameters in our algorithm. Our numerical experiments on real-world
datasets show the effectiveness of the proposed model and algorithm. Moreover,
the code is available at https://github.com/nothing2wang/NMD-TM.