Sparse-Input Neural Network using Group Concave Regularization
Bin Luo Susan Halabi
Simultaneous feature selection and non-linear function estimation are
challenging, especially in high-dimensional settings where the number of
variables exceeds the available sample size in modeling. In this article, we
investigate the problem of feature selection in neural networks. Although the
group LASSO has been utilized to select variables for learning with neural
networks, it tends to select unimportant variables into the model to compensate
for its over-shrinkage. To overcome this limitation, we propose a framework of
sparse-input neural networks using group concave regularization for feature
selection in both low-dimensional and high-dimensional settings. The main idea
is to apply a proper concave penalty to the $l_2$ norm of weights from all
outgoing connections of each input node, and thus obtain a neural net that only
uses a small subset of the original variables. In addition, we develop an
effective algorithm based on backward path-wise optimization to yield stable
solution paths, in order to tackle the challenge of complex optimization
landscapes. Our extensive simulation studies and real data examples demonstrate
satisfactory finite sample performances of the proposed estimator, in feature
selection and prediction for modeling continuous, binary, and time-to-event
outcomes.