Introduction {#sec:intro}
============

Background {#sec:background}
==========

Learning Variable Importance with Theoretical Guarantee {#sec:theory}
=======================================================

Experiment Analysis {#sec:exp}
===================

Posterior Concentration and Uncertainty Quantification {#sec:exp_learning}
------------------------------------------------------

Effectiveness in High-dimensional Variable Selection {#sec:exp_selection}
----------------------------------------------------

Discussion and Future Directions {#sec:discussion}
================================

In this work, we investigate the theoretical basis underlying the deep ’s ability to achieve rigorous uncertainty quantification in variable selection. Using the square integrated gradient $\psi_p(f)=||\deriv{x_p}f||_n^2$ as the measure of variable importance, we established two new Bayesian nonparametric results on the ’s ability to learn and quantify uncertainty about variable importance. Our results suggest that the neural network can learn variable importance effectively in high dimensions (Theorem \[thm:post\_convergence\]), in a speed that in some cases “breaks" the curse of dimensionality (Proposition \[thm:f\_post\_converg\]). Moreover, it can generate rigorous and calibrated uncertainty estimates in the sense that its $(1-q)$-level credible intervals for variable importance cover the true parameter $(1-q)\%$ of the time (Theorem \[thm:bvm\]-\[thm:bvm\_mvn\]). The simulation experiments confirmed these theoretical findings, and revealed the interesting fact that can learn variable importance $\psi_p(f)$ at a rate much faster than learning predictions for $f^*$ (Figure \[fig:learning\]). The comparative study illustrates the effectiveness of the proposed approach for the purpose of variable selection in high dimensions, which is a scenario where the existing methods experience difficulties due to model misspecification, the curse of dimensionality, or the issue of multiple comparisons. Consistent with classic Bayesian nonparametric and deep learning literature [@castillo_bernsteinvon_2015; @rockova_theory_2018; @barron_universal_1993; @barron_approximation_2018], the theoretical results developed in this work assume a well-specified scenario where $f^* \in \Fsc$ and the noise distribution is known. Furthermore, computing exact credible intervals under a model requires the use of MCMC procedures, which can be infeasible for large datasets. Therefore two important future directions of this work are to investigate the ’s ability to learn variable importance under model misspecification, and to identify posterior inference methods (e.g., particle filter [@dai_provable_2016] or structured variational inference [@pradier_projected_2018]) that are scalable to large datasets and can also achieve rigorous uncertainty quantification.

Acknowledgements {#acknowledgements .unnumbered}
================

I would like to thank Dr. Rajarshi Mukherjee and my advisor Dr. Brent Coull at Harvard Biostatistics for the helpful discussions and generous support. This publication was made possible by USEPA grant RD-83587201 Its contents are solely the responsibility of the grantee and does not necessarily represent the official views of the USEPA. Further, USEPA does not endorse the purchase of any commercial products or services mentioned in the publication.

[10]{}

M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng. : [A]{} system for large-scale machine learning. , May 2016. arXiv: 1605.08695.

R. A. Adams and J. J. F. Fournier. . Academic Press, Amsterdam, 2 edition edition, July 2003.

A. Altmann, L. Toloşi, O. Sander, and T. Lengauer. Permutation importance: a corrected feature importance measure. , 26(10):1340–1347, May 2010.

U. Anders and O. Korn. Model selection in neural networks. , 12(2):309–323, Mar. 1999.

C. Andrieu and J. Thoms. A tutorial on adaptive [MCMC]{}. , 18(4):343–373, Dec. 2008.

F. Bach. Breaking the [Curse]{} of [Dimensionality]{} with [Convex]{} [Neural]{} [Networks]{}. , Dec. 2014. arXiv: 1412.8690.

R. F. Barber and E. J. Candès. Controlling the false discovery rate via knockoffs. , 43(5):2055–2085, Oct. 2015.

A. R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. , 39(3):930–945, May 1993.

A. R. Barron and J. M. Klusowski. Approximation and [Estimation]{} for [High]{}-[Dimensional]{} [Deep]{} [Learning]{} [Networks]{}. , Sept. 2018. arXiv: 1809.03090.

Y. Benjamini and Y. Hochberg. Controlling the [False]{} [Discovery]{} [Rate]{}: [A]{} [Practical]{} and [Powerful]{} [Approach]{} to [Multiple]{} [Testing]{}. , 57(1):289–300, 1995.

L. Breiman. Random [Forests]{}. , 45(1):5–32, Oct. 2001.

P. Bühlmann. Statistical significance in high-dimensional linear models. , 19(4):1212–1242, Sept. 2013.

E. Candès, Y. Fan, L. Janson, and J. Lv. Panning for gold: ‘model-[X]{}’ knockoffs for high dimensional controlled variable selection. , 80(3), June 2018.

G. Castellano and A. M. Fanelli. Variable selection using neural-network models. , 31(1):1–13, Mar. 2000.

I. Castillo and J. Rousseau. A [Bernstein]{}–von [Mises]{} theorem for smooth functionals in semiparametric models. , 43(6):2353–2383, Dec. 2015.

B. Dai, N. He, H. Dai, and L. Song. Provable [Bayesian]{} [Inference]{} via [Particle]{} [Mirror]{} [Descent]{}. In [*Artificial [Intelligence]{} and [Statistics]{}*]{}, pages 985–994, May 2016.

J. Feng and N. Simon. Sparse [Input]{} [Neural]{} [Networks]{} for [High]{}-dimensional [Nonparametric]{} [Regression]{} and [Classification]{}. , Nov. 2017. arXiv: 1711.07592.

A. Gelman, J. Hill, and M. Yajima. Why [We]{} ([Usually]{}) [Don]{}’t [Have]{} to [Worry]{} [About]{} [Multiple]{} [Comparisons]{}. , 5(2):189–211, Apr. 2012.

S. Ghosal and A. van der Vaart. Convergence rates of posterior distributions for noniid observations. , 35(1):192–223, Feb. 2007.

S. Ghosh and F. Doshi-Velez. Model [Selection]{} in [Bayesian]{} [Neural]{} [Networks]{} via [Horseshoe]{} [Priors]{}. , May 2017. arXiv: 1705.10388.

F. Giordano, M. L. Rocca, and C. Perna. Input [Variable]{} [Selection]{} in [Neural]{} [Network]{} [Models]{}. , 43(4):735–750, Feb. 2014.

R. Gribonval, G. Kutyniok, M. Nielsen, and F. Voigtlaender. Approximation spaces of deep neural networks. , May 2019. arXiv: 1905.01208.

I. Guyon and A. Elisseeff. An [Introduction]{} to [Variable]{} and [Feature]{} [Selection]{}. , 3(Mar):1157–1182, 2003.

K. He, X. Zhang, S. Ren, and J. Sun. Deep [Residual]{} [Learning]{} for [Image]{} [Recognition]{}. In [*2016 [IEEE]{} [Conference]{} on [Computer]{} [Vision]{} and [Pattern]{} [Recognition]{} ([CVPR]{})*]{}, pages 770–778, June 2016.

X. He, J. Wang, and S. Lv. Scalable kernel-based variable selection with sparsistency. , Feb. 2018. arXiv: 1802.09246.

A. Krizhevsky, I. Sutskever, and G. E. Hinton. with [Deep]{} [Convolutional]{} [Neural]{} [Networks]{}. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, [*Advances in [Neural]{} [Information]{} [Processing]{} [Systems]{} 25*]{}, pages 1097–1105. Curran Associates, Inc., 2012.

M. La Rocca and C. Perna. Variable selection in neural network regression models with dependent data: a subsampling approach. , 48(2):415–429, Feb. 2005.

Y. LeCun, J. S. Denker, and S. A. Solla. Optimal [Brain]{} [Damage]{}. In D. S. Touretzky, editor, [*Advances in [Neural]{} [Information]{} [Processing]{} [Systems]{} 2*]{}, pages 598–605. Morgan-Kaufmann, 1990.

H. K. Lee. Consistency of posterior distributions for neural networks. , 13(6):629–642, July 2000.

F. Liang, Q. Li, and L. Zhou. Bayesian [Neural]{} [Networks]{} for [Selection]{} of [Drug]{} [Sensitive]{} [Genes]{}. , 113(523):955–972, July 2018.

R. Lockhart, J. Taylor, R. Tibshirani, and R. Tibshirani. A significance test for the [LASSO]{}. , 42, Jan. 2013.

C. Louizos, M. Welling, and D. P. Kingma. Learning [Sparse]{} [Neural]{} [Networks]{} through [L]{}\_0 [Regularization]{}. Feb. 2018.

Y. Y. Lu, Y. Fan, J. Lv, and W. S. Noble. : reproducible feature selection in deep neural networks. Sept. 2018.

R. May, G. Dandy, and H. Maier. Review of [Input]{} [Variable]{} [Selection]{} [Methods]{} for [Artificial]{} [Neural]{} [Networks]{}. , Apr. 2011.

H. Montanelli and Q. Du. New error bounds for deep networks using sparse grids. , Dec. 2017. arXiv: 1712.08688.

R. M. Neal. . Lecture [Notes]{} in [Statistics]{}. Springer-Verlag, New York, 1996.

M. F. Pradier, W. Pan, J. Yao, S. Ghosh, and F. Doshi-velez. Projected [BNNs]{}: [Avoiding]{} weight-space pathologies by learning latent representations of neural network weights. , Nov. 2018. arXiv: 1811.07006.

V. Rockova and N. Polson. Posterior [Concentration]{} for [Sparse]{} [Deep]{} [Learning]{}. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, [*Advances in [Neural]{} [Information]{} [Processing]{} [Systems]{} 31*]{}, pages 930–941. Curran Associates, Inc., 2018.

V. Rockova and E. Saha. On [Theory]{} for [BART]{}. , Oct. 2018. arXiv: 1810.00787.

L. Rosasco, S. Villa, S. Mosci, M. Santoro, and A. Verri. Nonparametric [Sparsity]{} and [Regularization]{}. , 14:1665–1714, 2013.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. . , 115(3):211–252, Dec. 2015.

S. Scardapane, D. Comminiello, A. Hussain, and A. Uncini. Group sparse regularization for deep neural networks. , 241:81–89, June 2017.

J. Schmidt-Hieber. Nonparametric regression using deep neural networks with [ReLU]{} activation function. , Aug. 2017. arXiv: 1708.06633.

K. Simonyan and A. Zisserman. Very [Deep]{} [Convolutional]{} [Networks]{} for [Large]{}-[Scale]{} [Image]{} [Recognition]{}. , Sept. 2014. arXiv: 1409.1556.

T. Suzuki. Adaptivity of deep [ReLU]{} network for learning in [Besov]{} and mixed smooth [Besov]{} spaces: optimal rate and curse of dimensionality. Sept. 2018.

C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the [Inception]{} [Architecture]{} for [Computer]{} [Vision]{}. , pages 2818–2826, 2015.

R. Tibshirani. Regression [Shrinkage]{} and [Selection]{} via the [Lasso]{}. , 58(1):267–288, 1996.

A. W. van der Vaart. . Cambridge University Press, Cambridge, June 2000.

G. Wahba. . SIAM, Sept. 1990. Google-Books-ID: ScRQJEETs0EC.

H. White and J. Racine. Statistical inference, the bootstrap, and neural-network modeling with application to foreign exchange rates. , 12(4):657–673, July 2001.

L. Yang, S. Lv, and J. Wang. Model-free [Variable]{} [Selection]{} in [Reproducing]{} [Kernel]{} [Hilbert]{} [Space]{}. , 17(82):1–24, 2016.

D. Yarotsky. Error bounds for approximations with deep [ReLU]{} networks. , Oct. 2016. arXiv: 1610.01145.

M. Ye and Y. Sun. Variable [Selection]{} via [Penalized]{} [Neural]{} [Network]{}: a [Drop]{}-[Out]{}-[One]{} [Loss]{} [Approach]{}. In [*International [Conference]{} on [Machine]{} [Learning]{}*]{}, pages 5620–5629, July 2018.

Statement of Multivariate Theorem {#sec:bvm_mvn}
=================================

For $f \in \Fsc(L, W, S, B)$, assuming the posterior distribution $\Pi_n(f)$ contracts around $f_0$ at rate $\epsilon_n$. Denote $\hat{\psi}^c=[\hat{\psi}^c_{1}, \dots, \hat{\psi}^c_P]$ for $\hat{\psi}^c_p$ as defined in Theorem \[thm:bvm\]. Also recall that $P=o(1)$, i.e. the data dimension does not grow with sample size.

Then $\hat{\psi}^c$ is an unbiased estimator of $\psi(f_0)=[\psi_1(f_0), \dots, \psi_P(f_0)]$, and the posterior distribution for $\psi^c(f)$ asymptotically converge toward a multivariate normal distribution surrounding $\hat{\psi}^c$, i.e. $$\begin{aligned}
\Pi\Big(\sqrt{n}(\psi^c(f) - \hat{\psi}^c) 
\Big| \{\bx_i, y_i\}_{i=1}^n \Big) \leadsto MVN(0, V_0),
%\label{eq:mvn_bvm}\end{aligned}$$ where $V_0$ is a $P \times P$ matrix such that $(V_0)_{p_1, p_2} =  4\inner{H_{p_1}f_0}{H_{p_2}f_0}_n$. \[thm:bvm\_mvn\]

Proof is in Supplementary Section \[sec:bvm\_mvn\_proof\].

Table and Figures
=================
