kNN Algorithm for Conditional Mean and Variance Estimation with
  Automated Uncertainty Quantification and Variable Selection
Marcos Matabuena Juan C. Vidal Oscar Hernan Madrid Padilla Jukka-Pekka Onnela
In this paper, we introduce a kNN-based regression method that synergizes the
scalability and adaptability of traditional non-parametric kNN models with a
novel variable selection technique. This method focuses on accurately
estimating the conditional mean and variance of random response variables,
thereby effectively characterizing conditional distributions across diverse
scenarios.Our approach incorporates a robust uncertainty quantification
mechanism, leveraging our prior estimation work on conditional mean and
variance. The employment of kNN ensures scalable computational efficiency in
predicting intervals and statistical accuracy in line with optimal
non-parametric rates. Additionally, we introduce a new kNN semi-parametric
algorithm for estimating ROC curves, accounting for covariates. For selecting
the smoothing parameter k, we propose an algorithm with theoretical
guarantees.Incorporation of variable selection enhances the performance of the
method significantly over conventional kNN techniques in various modeling
tasks. We validate the approach through simulations in low, moderate, and
high-dimensional covariate spaces. The algorithm's effectiveness is
particularly notable in biomedical applications as demonstrated in two case
studies. Concluding with a theoretical analysis, we highlight the consistency
and convergence rate of our method over traditional kNN models, particularly
when the underlying regression model takes values in a low-dimensional space.