Mini-Hes: A Parallelizable Second-order Latent Factor Analysis Model
Jialiang Wang Weiling Li Yurong Zhong Xin Luo
Interactions among large number of entities is naturally high-dimensional and
incomplete (HDI) in many big data related tasks. Behavioral characteristics of
users are hidden in these interactions, hence, effective representation of the
HDI data is a fundamental task for understanding user behaviors. Latent factor
analysis (LFA) model has proven to be effective in representing HDI data. The
performance of an LFA model relies heavily on its training process, which is a
non-convex optimization. It has been proven that incorporating local curvature
and preprocessing gradients during its training process can lead to superior
performance compared to LFA models built with first-order family methods.
However, with the escalation of data volume, the feasibility of second-order
algorithms encounters challenges. To address this pivotal issue, this paper
proposes a mini-block diagonal hessian-free (Mini-Hes) optimization for
building an LFA model. It leverages the dominant diagonal blocks in the
generalized Gauss-Newton matrix based on the analysis of the Hessian matrix of
LFA model and serves as an intermediary strategy bridging the gap between
first-order and second-order optimization methods. Experiment results indicate
that, with Mini-Hes, the LFA model outperforms several state-of-the-art models
in addressing missing data estimation task on multiple real HDI datasets from
recommender system. (The source code of Mini-Hes is available at
https://github.com/Goallow/Mini-Hes)