Nonparametric Estimation via Variance-Reduced Sketching
Yuehaw Khoo Yifan Peng Daren Wang
Nonparametric models are of great interest in various scientific and
engineering disciplines. Classical kernel methods, while numerically robust and
statistically sound in low-dimensional settings, become inadequate in
higher-dimensional settings due to the curse of dimensionality. In this paper,
we introduce a new framework called Variance-Reduced Sketching (VRS),
specifically designed to estimate density functions and nonparametric
regression functions in higher dimensions with a reduced curse of
dimensionality. Our framework conceptualizes multivariable functions as
infinite-size matrices, and facilitates a new sketching technique motivated by
numerical linear algebra literature to reduce the variance in estimation
problems. We demonstrate the robust numerical performance of VRS through a
series of simulated experiments and real-world data applications. Notably, VRS
shows remarkable improvement over existing neural network estimators and
classical kernel methods in numerous density estimation and nonparametric
regression models. Additionally, we offer theoretical justifications for VRS to
support its ability to deliver nonparametric estimation with a reduced curse of
dimensionality.