Semi-Supervised Deep Sobolev Regression: Estimation, Variable Selection
  and Beyond
Zhao Ding Chenguang Duan Yuling Jiao Jerry Zhijian Yang
We propose SDORE, a semi-supervised deep Sobolev regressor, for the
nonparametric estimation of the underlying regression function and its
gradient. SDORE employs deep neural networks to minimize empirical risk with
gradient norm regularization, allowing computation of the gradient norm on
unlabeled data. We conduct a comprehensive analysis of the convergence rates of
SDORE and establish a minimax optimal rate for the regression function.
Crucially, we also derive a convergence rate for the associated plug-in
gradient estimator, even in the presence of significant domain shift. These
theoretical findings offer valuable prior guidance for selecting regularization
parameters and determining the size of the neural network, while showcasing the
provable advantage of leveraging unlabeled data in semi-supervised learning. To
the best of our knowledge, SDORE is the first provable neural network-based
approach that simultaneously estimates the regression function and its
gradient, with diverse applications including nonparametric variable selection
and inverse problems. The effectiveness of SDORE is validated through an
extensive range of numerical simulations and real data analysis.