---
abstract: |
    We study the finite sample behavior of Lasso-based inference methods such as post double Lasso and debiased Lasso. Empirically and theoretically, we show that these methods can exhibit substantial omitted variable biases (OVBs) due to Lasso not selecting relevant controls. This phenomenon can be systematic in finite samples and occur even when the coefficients are very sparse and the sample size is large and larger than the number of controls. Therefore, relying on the existing asymptotic inference theory can be problematic in empirical applications. We compare the Lasso-based inference methods to modern high-dimensional OLS-based methods and provide practical guidance.

    **Keywords:** Lasso, post double Lasso, debiased Lasso, OLS, omitted variable bias, limited variability, finite sample analysis
author:
- 'Kaspar Wüthrich[^1] Ying Zhu[^2]\'
bibliography:
- 'bibliography.bib'
date: 'First draft on arXiv: March 20, 2019. This draft: .'
title: 'Omitted variable bias of Lasso-based inference methods: A finite sample analysis[^3] '
---

Introduction
============

Researchers are often interested in making statistical inferences on a single parameter (for example, the effect of a treatment or a policy), while controlling for confounding factors. In more and more economic applications, the number of potential control variables ($p$) is becoming large relative to the sample size ($n$), either due to the inherent richness of the data, the desire of researchers to specify flexible functional forms, or both. In such problems, a natural approach is to use the least absolute shrinkage and selection operator (Lasso), introduced by @tibsharini1996regression, to select the relevant controls (i.e., those with nonzero coefficients) and then run OLS with the selected controls. However, this approach has been criticized because, unless the magnitude of the coefficients associated with the relevant controls is very small, it requires these coefficients to be well separated from zero to ensure that Lasso selects them. This critique has motivated the development of post double Lasso [@belloni2014inference] and debiased Lasso [@javanmard2014confidence; @vandergeer2014asymptotically; @Zhang_Zhang]. The breakthrough in this literature is that it does not require the aforementioned separation condition, and the Lasso not selecting relevant controls yields negligible asymptotic biases under certain conditions on $n$, $p$, and the degree of sparsity.

Since their introduction, post double Lasso and debiased Lasso have quickly become the most popular inference methods for problems with many control variables. Given the rapidly growing (asymptotic) theoretical and applied literature on these methods, it is crucial to take a step back and examine the performance of these new procedures in empirically relevant settings as well as to better understand their merits and limitations relative to other alternatives. In particular, there is a misconception that the post double Lasso and debiased Lasso are immune to under-selection of the Lasso because they do not require the above-mentioned separation condition. Empirically and theoretically, this paper shows that, in finite samples, under-selection can result in substantial OVBs of these methods and yield invalid inferences. We also compare the post double Lasso and debiased Lasso to modern high-dimensional OLS-based inference procedures.

Let us consider the linear model $$\begin{aligned}
Y_{i} & = & D_{i}\alpha^{*}+X_{i}\beta^{*}+\eta_{i},\label{eq:main-y}\\
D_{i} & = & X_{i}\gamma^{*}+v_{i}.\label{eq:main-d}\end{aligned}$$ Here $Y_{i}$ is the outcome, $D_{i}$ is the scalar treatment variable of interest, and $X_{i}$ is a $(1\times p)$-dimensional vector of additional control variables. To focus on the impact of under-selection and facilitate the exposition, we assume that (\[eq:main-y\]) and (\[eq:main-d\]) share the same set of $k$ non-zero coefficients. In this paper, we study the performance of post double Lasso and the debiased Lasso for estimating and making inferences (e.g., constructing confidence intervals) on the treatment effect $\alpha^{\ast}$.

We present extensive simulation evidence demonstrating that post double Lasso and debiased Lasso can exhibit substantial OVBs relative to the standard deviations due to the Lasso not selecting all the relevant controls. Our simulation results can be summarized as follows. (i) Large OVBs are persistent across a range of empirically relevant settings and can occur even when $n$ is large and larger than $p$, and $k$ is small (e.g., when $n=10000$, $p=4000$, $k=5$). (ii) For the same $(n,p,k)$, noise variances, and magnitude of coefficients, there can be no OVBs at all, small OVBs, or substantial OVBs, depending on the variance of the relevant controls. (iii) When the controls exhibit limited variability, the performance of Lasso-based inference methods can be very sensitive to the choice of the regularization parameters; under sufficient variability, post double Lasso is less sensitive. (iv) There is no simple recommendation for how to choose the regularization parameters.[^4] (v) The OVBs can lead to invalid inferences and under-coverage of confidence intervals.

In addition to the simulations, we conduct Monte Carlo studies based on two empirical applications: The analysis of the effect of 401(k) plans on savings by @belloni2017program and the study of the racial test score gap by @fryerlevitt2013. We draw samples of different size from the large original datasets and compare the subsample estimates to the estimates based on the original data. This exercise mimics random sampling from a large super-population. In both applications, we find substantial biases even when $n$ is considerably larger than $p$, and document that the magnitude of the biases varies substantially depending on the regularization choice.

The existing (asymptotic) theory provides little insight about the OVBs of the Lasso-based inference methods documented in our simulation studies. In terms of formal results, it only implies an upper bound of $\texttt{constant}\cdot\frac{k\log p}{n}$ for the bias. Here the (positive) $\texttt{constant}$ does not depend on $(n,p,k)$ and bears little meaning in the existing theory which simply assumes $\frac{k\log p}{\sqrt{n}}\rightarrow0$ among other sufficient conditions.[^5] The asymptotic upper bound $\texttt{constant}\cdot\frac{k\log p}{n}$ is only informative about the *least favorable* case and does not explain the following practically relevant questions: (i) When do OVBs arise? (ii) Why can the OVBs be drastically different despite $(n,p,k)$, noise variances, and absolute values of coefficients being the same? (iii) What is the magnitude of OVBs in the *most favorable* cases? (iv) How severe can the OVBs be in finite samples where $\frac{k\log p}{n}$ is not small enough? To explain (i) and (ii), we provide theoretical conditions under which the OVBs occur systematically and establish a novel result on the under-selection of the Lasso. To explain (iii) and (iv), we derive new informative lower and upper bounds on the OVBs of post double Lasso and the debiased Lasso proposed by @vandergeer2014asymptotically. Our analyses are non-asymptotic and allow us to study the OVBs for fixed $(n,p,k)$, but are also informative when $\frac{k\log p}{n}\rightarrow0$ or $\frac{k\log p}{n}\rightarrow\infty$. Our theoretical results reveal that, in finite samples, the OVBs are not just simple linear functions of $\frac{k\log p}{n}$ but depend on $n$, $p$, and $k$ in a more complex way. In one of our results, we derive explicit universal constants, allowing us to compute precise lower bounds and perform “comparative statics” given features of the underlying empirical problems. This lower bound analysis of the OVBs is the first of its kind in the literature. In contrast to upper bound analyses, it is informative about the most favorable cases and thus the finite sample limitations of Lasso-based inference methods. Our results suggest that the OVBs can be substantial relative to the standard deviation derived from the existing theory. As a consequence, the confidence intervals proposed in the literature can exhibit under-coverage.

In the main part of the paper, we focus on post double Lasso and present results for the debiased Lasso in the appendix. Post double Lasso consists of two Lasso selection steps: A Lasso regression of $Y_{i}$ on $X_{i}$ and a Lasso regression of $D_{i}$ on $X_{i}$. In the third step, the estimator of $\alpha^{\ast}$, $\tilde{\alpha}$, is the OLS regression of $Y_{i}$ on $D_{i}$ and the union of controls selected in the two Lasso steps. For the setup of (\[eq:main-y\])–(\[eq:main-d\]), post double has a clear advantage over a post (single) Lasso OLS. As @belloni2014inference [p.614] put it: “Intuitively, this procedure \[post double Lasso\] works well since we are more likely to recover key controls by considering selection of controls from both equations instead of just considering selection of controls from the single equation”. However, OVBs can still arise whenever the relevant controls are selected in neither Lasso steps. To formally study the OVBs, one has to first understand when such “double under-selection” is likely to occur in finite samples. This task is difficult because it requires necessary results on the Lasso’s inclusion to show that double under-selection can occur with high probability.

To our knowledge, no formal results exist that can explain the phenomenon “double under-selection with high probability”, and this is likely why the literature provides no characterization of the finite sample OVBs. In this paper, under some classical assumptions, we prove that if the ratios of the absolute values of the non-zero coefficients and the variances of the controls are no greater than half the regularization parameters[^6] (or, in other words, if the products of the absolute values of the non-zero coefficients and the standard deviations of the corresponding controls are small relative to the noise’s standard deviation), Lasso fails to select these controls in both steps with high probability.

Our new result on the under-selection of the Lasso is the key ingredient for deriving a lower bound formula for the OVB of the post double Lasso estimator $\tilde{\alpha}$. The OVB lower bound is characterized by the complicated interplay between the probability of double under-selection and the omitted coefficients, and is not a simple linear function of $\frac{k\log p}{n}$ in general. It can be substantial compared to the standard deviation obtained from the asymptotic distribution in @belloni2014inference[^7], even in settings where $k$ is very small, $n$ is large and larger than $p$. As $\frac{k\log p}{n}$ gets large, depending on the configuration of $\left(\beta_{K}^{*},\,\gamma_{K}^{*},\,\alpha^{*}\right)$, the OVB lower bounds scale as $\left|\alpha^{*}\right|$ or as $\frac{\sigma_{\eta}}{\sigma_{v}}$, where $\sigma_{\eta}^{2}$ is the variance of the $\eta_{i}$s in (\[eq:main-y\]) and $\sigma_{v}^{2}$ is the variance of the $v_{i}$s in (\[eq:main-d\]). This raises the question of how severe the OVBs can be when $\frac{k\log p}{n}\rightarrow\infty$ and both Lasso steps are inconsistent in terms of the prediction errors (and $\ell_{2}$-errors). To answer this question, we would like a meaningful upper bound (other than just $\infty$) on the OVBs, but there is no such result in the existing literature. Interestingly enough, under this undesirable regime, we show that the upper bounds on OVBs scale as $\left|\alpha^{*}\right|$ or $\frac{\sigma_{\eta}}{\sigma_{v}}$ with high probability.

The occurrence of double under-selection and the magnitude of OVBs both depend on the absolute values of the non-zero coefficients and the standard deviations of the corresponding controls in both Lasso steps through their products. Therefore, under-selection cannot be avoided or mitigated by rescaling the covariates and the OVB lower bound remains the same after rescaling. Any normalization of $X_{i}$ simply leads to rescaled coefficients and vice versa, while their products stay the same. This result suggests an equivalence between “small” (nonzero) coefficient problems and problems with “limited” variability in the relevant controls. By rescaling the controls, the former can always be recast as the latter and vice versa.

Given our theoretical results, all else equal, limited variability in the relevant controls makes it more likely for the Lasso to omit them. Limited variability is ubiquitous in applied economic research and there are many instances where it occurs by design. First, it naturally arises from small cells (i.e., when there are only a few observations in some of the cells defined by covariate values). Small cells are prevalent in specifications that include many two-way interactions and are saturated in at least a subset of controls.[^8] When the controls are discrete, limited overlap a major concern in research designs relying on unconfoundedness-type identification assumptions can be viewed as a small cell problem [e.g., @rothe2017robust]. Moreover, categorical controls, when incorporated through a set of indicator variables, give rise to small cells if some of the categories are sparsely populated. Second, when researchers perform subsample analyses, there are often controls that exhibit limited variability within subsamples. Third, in times series and “large $T$” panel data applications, persistence in the controls over time can lead to limited variability. Finally, many empirical settings feature high-dimensional fixed effects that suffer from limited variability. Some authors propose to penalize the fixed effects [e.g., @kock2019uniform] with the Lasso regularization, while others do not [e.g., @belloni2016cluster]. The results in this paper suggest that penalizing fixed effects with the Lasso regularization can be problematic.

Our theoretical analyses of the OVBs of the post double Lasso have important implications for inference based on the procedure proposed in @belloni2014inference. On the one hand, @belloni2014inference show that the assumptions $\frac{k\log p}{\sqrt{n}}\rightarrow0$, $\left|\alpha^{*}\right|$ being bounded from above, and $\left(\sigma_{\eta},\,\sigma_{v}\right)$ being bounded away from zero and above (among other regularity conditions) are *sufficient* for establishing the asymptotic normality and unbiasedness of their post double Lasso procedure, regardless of whether under-selection is present or not. On the other hand, we show empirically relevant settings where double under-selection occurs with high probability, and also show that in such settings, small $\frac{k\log p}{\sqrt{n}}$ and $\frac{\left|\alpha^{*}\right|\sigma_{v}}{\sigma_{\eta}}$ are *almost necessary* for a good performance of the post double Lasso. Unfortunately in practice, because $k$ and $\alpha^{*}$ are fundamentally unknown, it is impossible to know whether or not $\frac{k\log p}{\sqrt{n}}$ and $\frac{\left|\alpha^{*}\right|\sigma_{v}}{\sigma_{\eta}}$ are small enough. Moreover, the requirement of small $\frac{k\log p}{\sqrt{n}}$ is actually quite demanding in high dimensional settings. For example, even with $n=10000>p=4000$ and $k=5$, we have $\frac{k\log p}{\sqrt{n}}\approx0.41$.

In view of our theoretical results, simulations, and empirical evidence, a natural question is how to make statistical inference in a reliable manner, especially when limited variability is present. The case where $p$ is comparable to but still smaller than $n$ deserves special attention because of its prevalence in applied economic research.[^9] In this case, recently developed high-dimensional OLS-based inference methods [e.g., @cattaneo2018inference; @jochmans2018heteroscedasticity; @kline2018leaveout; @dadamo2018cluster] constitute a natural alternative to Lasso-based procedures. These modern OLS-based methods are based on (unbiased) OLS estimators and variance estimators that are robust to the inclusion of many controls.[^10] Based on extensive simulations, we find that OLS with standard errors proposed by @cattaneo2018inference is unbiased (as expected) and demonstrates excellent size accuracy, irrespective of the degree of variability in the controls. Another advantage of OLS-based methods is that, unlike the Lasso-based inference methods, they do not rely on any sparsity assumptions. This is important because sparsity assumptions may not be satisfied in applications and, as we show in this paper, the OVBs of the Lasso-based inference procedures can be substantial even when $k$ is very small, $n$ is very large and larger than $p$. However, OLS yields somewhat wider confidence intervals than the Lasso-based inference methods, which suggests that there is a trade-off between coverage accuracy and the width of the confidence intervals.

Our analyses suggest several recommendations concerning the use of Lasso-based inference methods in empirical studies. First, one should always perform robustness checks with respect to the choice of the regularization parameters. Our simulation and theoretical results suggest the following heuristic: If the estimates of $\alpha^{*}$ are robust to increasing the theoretically recommended regularization parameters in the two Lasso steps, post double Lasso could be a reliable and efficient method. Second, our findings highlight the usefulness of augmenting the final OLS regression in post double Lasso with control variables motivated by economic theory and prior knowledge, as suggested by @belloni2014inference. Third, when $p$ is smaller than $n$, modern high-dimensional OLS-based inference methods constitute a viable alternative to Lasso-based inference methods.

Our paper highlights the intrinsic limitations of basing inference procedures on variable selection in empirically relevant settings, and opens up interesting avenues for future research. For instance, we expect under-selection to have important implications for the finite sample behavior of many other linear and nonlinear inference procedures based on the Lasso [e.g., @belloni2012sparse; @belloni2014inference; @farrell2015robust; @belloni2017program; @chernozhukov2018double]. Furthermore, our paper motivates systematic analyses of the practical usefulness of Lasso-based inference procedures and other modern high-dimensional methods. For example, @angrist2019machine investigate the usefulness of post double Lasso for regression-based sensitivity analyses, and present simulation evidence on the finite sample behavior of Lasso-based instrumental variables (IV) methods.

Finally, we emphasize that our results do not contradict the existing asymptotic theory on post double Lasso and the debiased Lasso, but rather offer a better understanding of when these methods perform well in finite samples and when they do not. This paper should not be viewed as a criticism of these methods and there is no doubt that they provide significant theoretical improvements over earlier procedures (e.g., post single Lasso OLS). Instead, our findings should be viewed as a warning that relying on the existing asymptotic inference theory for post double Lasso and debiased Lasso can be problematic in empirical applications and these methods can still suffer from the consequence of under-selection in finite samples.

Lasso and post double Lasso {#sec:lasso_post_double_lassp}
===========================

The Lasso {#sec:lasso}
---------

Consider the following linear regression model $$Y_{i}=X_{i}\theta^{*}+\varepsilon_{i},\qquad i=1,\dots,n,\label{eq:1}$$ where $\left\{ Y_{i}\right\} _{i=1}^{n}=Y$ is an $n$-dimensional response vector, $\left\{ X_{i}\right\} _{i=1}^{n}=X$ is an $n\times p$ matrix of covariates with $X_{i}$ denoting the $i$th row of $X$, $\left\{ \varepsilon_{i}\right\} _{i=1}^{n}=\varepsilon$ is a zero-mean error vector, and $\theta^{*}$ is a $p$-dimensional vector of unknown coefficients.

The Lasso estimator of $\theta^{*}$ is given by $$\hat{\theta}\in\arg\min_{\theta\in\mathbb{R}^{p}}\frac{1}{2n}\sum_{i=1}^{n}\left(Y_{i}-X_{i}\theta\right)^{2}+\lambda\sum_{j=1}^{p}\left|\theta_{j}\right|,\label{eq:las}$$ where $\lambda$ is the regularization parameter. For example, if $\varepsilon\sim\mathcal{N}\left(0_{n},\,\sigma^{2}I_{n}\right)$ and $X$ is a fixed design matrix with normalized columns (i.e., $\frac{1}{n}\sum_{i=1}^{n}X_{ij}^{2}=b$ for all $j=1,\dots,p$), @bickel2009simultaneous set $\lambda=2\sigma\sqrt{\frac{2b\left(1+\tau\right)\log p}{n}}$ (where $\tau>0$) to establish upper bounds on $\sqrt{\sum_{j=1}^{p}(\hat{\theta}_{j}-\theta_{j}^{*})^{2}}$ with high probability guarantee. @wainwright2009sharp sets $\lambda$ proportional to $\frac{\sigma}{\phi}\sqrt{\frac{b\log p}{n}}$, where $\phi\in(0,\,1]$ is a measure of correlation between the covariates with nonzero coefficients and those with zero coefficients, to establish perfect selection.

Besides the classical choices in @bickel2009simultaneous and @wainwright2009sharp mentioned above, other choices of $\lambda$ are available in the literature. For instance, @belloni2013least develop a data-dependent approach and @belloni2012sparse and @belloni2016cluster propose choices that accommodate heteroscedastic and clustered errors. In the case of nearly orthogonal $X$ (which is typically required to ensure a good performance of the Lasso), these choices of $\lambda$ have a similar magnitude as those in @bickel2009simultaneous and @wainwright2009sharp. Finally, a very popular practical approach for choosing $\lambda$ is cross-validation. However, only a few theoretical results exist on the properties of Lasso when $\lambda$ is chosen using cross-validation; see, for example, @homrighausen2013thelasso [@homrighausen2014leaveoneout] and @chetverikov2017cv.

Post double Lasso
-----------------

The model implies the following reduced form model for $Y_{i}$: $$\begin{aligned}
Y_{i} & = & X_{i}\pi^{*}+u_{i},\label{eq:reduce-1}\end{aligned}$$ where $\pi^{*}=\gamma^{*}\alpha^{*}+\beta^{*}$ and $u_{i}=\eta_{i}+\alpha^{*}v_{i}$.

The post double Lasso, introduced by @belloni2014inference, essentially exploits the Frisch-Waugh theorem, where the regressions of $Y$ on $X$ and $D$ on $X$ are implemented with the Lasso: $$\begin{aligned}
\hat{\pi} & \in & \textrm{arg}\min_{\pi\in\mathbb{R}^{p}}\frac{1}{2n}\sum_{i=1}^{n}\left(Y_{i}-X_{i}\pi\right)^{2}+\lambda_{1}\sum_{j=1}^{p}\left|\pi_{j}\right|,\label{eq:las-1}\\
\hat{\gamma} & \in & \textrm{arg}\min_{\gamma\in\mathbb{R}^{p}}\frac{1}{2n}\sum_{i=1}^{n}\left(D_{i}-X_{i}\gamma\right)^{2}+\lambda_{2}\sum_{j=1}^{p}\left|\gamma_{j}\right|.\label{eq:las-2}\end{aligned}$$ The final estimator $\tilde{\alpha}$ of $\alpha^{*}$ is then obtained from an OLS regression of $Y$ on $D$ and the union of selected controls $$\left(\tilde{\alpha},\,\tilde{\beta}\right)\in\textrm{arg}\min_{\alpha\in\mathbb{R},\beta\in\mathbb{R}^{p}}\frac{1}{2n}\sum_{i=1}^{n}\left(Y_{i}-D_{i}\alpha-X_{i}\beta\right)^{2}\quad\textrm{s.t. }\beta_{j}=0\;\forall j\notin\left\{ \hat{I}_{1}\cup\hat{I}_{2}\right\} ,\label{eq:double}$$ where $\hat{I}_{1}=\textrm{supp}\left(\hat{\pi}\right)=\left\{ j:\,\hat{\pi}_{j}\neq0\right\} $ and $\hat{I}_{2}=\textrm{supp}\left(\hat{\gamma}\right)=\left\{ j:\,\hat{\gamma}_{j}\neq0\right\} $.

Evidence on the OVB of post double Lasso {#sec:evidence}
========================================

This section presents evidence on the under-selection of the Lasso and the OVB of post double Lasso. Section \[sec:numerical\_evidence\] analyzes a simple numerical example, Section \[sec:simulation\_evidence\] presents simulation evidence for different choices of the regularization parameters, and Section \[sec:empirical\_evidence\] revisits two empirical applications.

In our simulations, we vary the variance of the control variables while fixing their coefficients. In particular, we consider settings where the control variables exhibit limited variability. It is important to note that all the results remain unchanged when we rescale the controls and transform the limited variability problem into a small coefficients problem; see Section \[sec:inference\] for a further discussion.

Numerical example {#sec:numerical_evidence}
-----------------

To study the under-selection of the Lasso, we simulate data according to the linear model , where $X_{i}\sim \mathcal{N}\left(0,\sigma_x^2I_{p}\right)$ is independent of $\varepsilon_{i}\sim\mathcal{N}(0,1)$ and $\left\{ X_{i},\varepsilon_{i}\right\} _{i=1}^{n}$ consists of i.i.d. entries. We set $n=500$, $p=200$, and consider a sparse setting where $\theta^{*}=(\underbrace{1,\dots,1}_{k},0,\dots,0)^{T}$ and $k=5$. We employ the classical recommendation for the regularization parameter by @bickel2009simultaneous.[^11]

Figure \[fig:performance\_illu\] displays the average number of selected (relevant and irrelevant) covariates as a function of the degree of variability, $\sigma_x$. The variability of the covariates significantly affects the selection performance of the Lasso. The average number of selected covariates is monotonically increasing in $\sigma_x$, ranging from approximately zero when $\sigma_x=0.05$ up to five when $\sigma_x=0.5$.

\[fig:performance\_illu\]

Next, we investigate the implications of the under-selection of Lasso for post double Lasso. We simulate data according to the structural model –, where $X_{i}\sim \mathcal{N}\left(0,\sigma_x^2I_{p}\right)$, $\eta_{i}\sim\mathcal{N}(0,1)$, and $v_{i}\sim \mathcal{N}(0,1)$ are independent of each other and $\left\{ X_{i},\eta_{i},v_{i}\right\} _{i=1}^{n}$ consists of i.i.d. entries. Our object of interest is $\alpha^{*}$. We set $n=500$, $p=200$, $\alpha^{*}=0$, and consider a sparse setting where $\beta^{*}=\gamma^{*}=(\underbrace{1,\dots,1}_{k},0,\dots,0)^{T}$ and $k=5$. In both Lasso steps, we employ the recommendation for the regularization parameter by @bickel2009simultaneous.

Figure \[fig:fsd\_dml\] displays the finite sample distribution of post double Lasso for different values of $\sigma_x$. For comparison, we plot the distribution of the “oracle estimator” of $\alpha^\ast$, a regression of $Y_i-X_i\pi^\ast$ on $D_i-X_i\gamma^\ast$.

\[fig:fsd\_dml\]

For $\sigma_x=0.05$, the post double Lasso estimator exhibits a small bias. Increasing $\sigma_x$ to $0.15$ shifts the distribution to the right, resulting in a large bias relative to the standard deviation. For $\sigma_x=0.3$, post double Lasso is biased and exhibits a larger standard deviation. Finally, for $\sigma_x=0.5$, the post double Lasso estimator is approximately unbiased and its distribution is centered at the true value $\alpha^{*}=0$. We emphasize that these striking differences arise despite $(n,p,k,\sigma_v,\sigma_\eta)$ and the coefficients being the same in these four designs.

The bias of post double Lasso is caused by the OVB arising from the two Lasso steps not selecting all the relevant covariates. Figure \[fig:hist\_sel\_dml\] displays the number of selected relevant covariates (i.e., the cardinality of $\hat{I}_{1}\cup\hat{I}_{2}$ in ).

\[fig:hist\_sel\_dml\]

With very high probability, none of the relevant controls get selected when $\sigma_x\le 0.15$. The selection performance improves as $\sigma_x$ increases, until, with high probability, all relevant regressors get selected when $\sigma_x=0.5$.[^12]

We further note that the shape of the finite sample distribution depends on $\sigma_x$. This distribution is a mixture of the distributions of OLS conditional on the two Lasso steps selecting different combinations of covariates.[^13] For $\sigma_x=0.05$, $\sigma_x=0.15$, and $\sigma_x=0.5$, the finite sample distributions are well-approximated by normal distributions (albeit centered at different values). The reason is that, when $\sigma_x=0.05$ or $\sigma_x=0.15$, none of the relevant controls get selected with high probability, and, when $\sigma_x=0.5$, the two Lasso steps almost always select all the relevant controls. In between these two cases, when $\sigma_x=0.3$, the finite sample distribution is a mixture of distributions with different means (depending on how many controls get selected) and has a larger standard deviation.

The event where none of the relevant covariates are selected is of particular interest. Figure \[fig:cond\_bias\] displays the probability of this event as a function of $\sigma_x$ as well as the OVB conditional on this event, which is computed as $$\frac{1}{\sum_{r=1}^{R}S_{r}}\sum_{r=1}^{R}\left(\tilde{\alpha}_{r}-\alpha^\ast\right)\cdot S_{r}.$$ In the above formula, $R$ is the total number of simulation repetitions, $S_{r}$ is an indicator which is equal to one if nothing gets selected and zero otherwise, and $\tilde{\alpha}_{r}$ is the estimate of $\alpha^\ast$ in the $r$[th]{} repetition. As the variability in the controls increases from $\sigma_x=0.05$ to $\sigma_x=0.4$, the probability that nothing gets selected is decreasing from one to zero. The OVB increases until $\sigma_x=0.35$ and is not defined for $\sigma_x>0.35$ because the empirical probability that nothing gets selected is zero in this case.

\[fig:cond\_bias\]

Importantly, the issue documented here is not a “small sample” phenomenon, but persists even in large sample settings. To illustrate, Figures \[fig:fsd\_large\] and \[fig:hist\_sel\_large\] display the finite sample distributions and the number of selected relevant covariates when $(n,p,k)=(10000,4000,5)$. Even in large samples, the finite sample distribution may not be centered at the true value and the bias can be large relative to the standard deviation because, with high probability, none of the relevant controls are selected.[^14] Compared to the results for $(n,p,k)=(500,200,5)$, under-selection and large biases occur at lower values of $\sigma_x$, and all the relevant controls get selected when $\sigma_x=0.15$.

\[fig:fsd\_large\]

\[fig:hist\_sel\_large\]

Simulation evidence {#sec:simulation_evidence}
-------------------

Section \[sec:numerical\_evidence\] considers a simple numerical example and the classical regularization choice by @bickel2009simultaneous based on the true $\sigma^2$. Here we present simulation evidence for three popular feasible choices of the regularization parameter: The heteroscedasticity-robust proposal in @belloni2012sparse ($\lambda_{\text{BCCH}}$)[^15], the regularization parameter with the minimum cross-validated error ($\lambda_{\text{min}}$), and the regularization parameter with the minimum cross-validation error plus one standard deviation ($\lambda_{\text{1se}}$).

We start by investigating the selection performance. We simulate data based on the DGP of Section \[sec:numerical\_evidence\], where $(n,p,k)=(500,200,5)$. To illustrate the impact of changing the sample size, we also show results for $n=1000$ and $n=200$. The results are based on 1,000 simulation repetitions. Figure \[fig:sel\] displays the average number of selected covariates as a function of $\sigma_x$. Lasso with $\lambda_{\text{BCCH}}$ selects the lowest number of covariates. Choosing $\lambda_{\text{1se}}$ leads to a somewhat higher number of selected covariates and results in moderate over-selection for larger values of $\sigma_x$. Lasso with $\lambda_{\text{min}}$ selects the highest number of covariates and exhibits substantial over-selection. Figure \[fig:sel\_rel\] shows the corresponding numbers of selected relevant covariates. We note that, when $\sigma_x=0.1$, the Lasso does not select all the relevant covariates even with $\lambda_{\text{min}}$ which can result in substantial over-selection. For all regularization choices, the selection performance improves as the sample size increases.

\[fig:sel\]

\[fig:sel\_rel\]

To investigate the impact of under-selection on post double Lasso, we simulate data according to the DGP of Section \[sec:numerical\_evidence\], where $(n,p,k)=(500,200,5)$, and also consider $n=1000$ and $n=200$. The results are based on 1,000 simulation repetitions. Appendix \[app:additional\_simulations\] presents additional simulation evidence, where we vary $k$, the distribution of $X_i$, the error terms $(\eta_i,v_i)$, the true value $\alpha^\ast$, and also consider a heteroscedastic DGP. We employ the same regularization choice in both Lasso steps.

Figure \[fig:dml\_bias\_std\] presents evidence on the bias of post double Lasso. To make the results easier to interpret, we report the ratio of the bias to the empirical standard deviation.

\[fig:dml\_bias\_std\]

Under limited variability, post double Lasso with $\lambda_{\text{BCCH}}$ can exhibit biases that are up to two times larger than the standard deviation when $n=200$ and still comparable to the standard deviation when $n=1000$. The relationship between $\sigma_x$ and the ratio of bias to standard deviation is non-monotonic: It is increasing for small $\sigma_x$ and decreasing for larger $\sigma_x$. The bias is somewhat smaller than the standard deviation when $\lambda=\lambda_{\text{1se}}$. Setting $\lambda=\lambda_{\text{min}}$ yields the smallest ratio of bias to standard deviation. For all choices of the regularization parameters, the ratio of bias to standard deviation is decreasing in the sample size. Finally, we note that when $\sigma_x$ is large enough such that there is no under-selection, post double Lasso performs well and is approximately unbiased for all regularization parameters.

The additional simulation evidence reported in Appendix \[app:additional\_simulations\] confirms these results and further shows that $\alpha^\ast$ is an important determinant of the performance of post double Lasso because of its direct effect on the magnitude of the coefficients and the error variance in the reduced form equation . Moreover, we show that, while choosing $\lambda=\lambda_{\text{min}}$ works well when $\alpha^\ast=0$, this choice can yield poor performances when $\alpha^\ast\ne 0$ (c.f. Figure \[fig:dml\_bias\_std\_app\], DGP A5).[^16] Thus, there is no simple recommendation for how to choose the regularization parameters in practice.

The substantive performance differences between the three regularization choices suggest that post double Lasso is sensitive to the regularization parameters when $\sigma_x$ is small (but not quite zero). To further investigate this issue, we compare the results for $\lambda_{\text{BCCH}}$, $0.5 \lambda_{\text{BCCH}}$, and $1.5 \lambda_{\text{BCCH}}$. Figures \[fig:sel\_sensitivity\] and \[fig:sel\_rel\_sensitivity\] display the average numbers of selected and selected relevant covariates. The differences in the selection performance are substantial. Lasso with $0.5 \lambda_{\text{BCCH}}$ over-selects for all $n$ and $\sigma_x$, while Lasso with $1.5 \lambda_{\text{BCCH}}$ under-selects unless $\sigma_x$ and $n$ are large. The differences get smaller as the sample size increases. Figure \[fig:dml\_bias\_std\_sens\] displays the ratio of bias and standard deviation for post double Lasso. Choosing $0.5 \lambda_{\text{BCCH}}$ yields small biases relative to the standard deviations for all $\sigma_x$. By contrast, choosing $1.5 \lambda_{\text{BCCH}}$ yields biases that can be more than four times larger than the standard deviations when $n=200$, and still be substantial when $n=1000$. For larger values of $\sigma_x$, post double Lasso is less sensitive to the choice of the regularization parameters. In Section \[sec:inference\], based on our theoretical results, we discuss how to interpret and use robustness checks with respect to the regularization parameters in empirical applications.

\[fig:sel\_sensitivity\]

\[fig:sel\_rel\_sensitivity\]

\[fig:dml\_bias\_std\_sens\]

In sum, our simulation evidence shows (i) that under-selection can lead to large biases relative to the standard deviation, (ii) that, under limited (but not quite zero) variability, the performance of post double Lasso is very sensitive to the choice of the regularization parameters, and (iii) that there is no simple recommendation for how to choose the regularization parameters in practice.

Empirical evidence {#sec:empirical_evidence}
------------------

### The effect of 401k plans on savings {#sec:401k}

Here we revisit the analysis of the causal effect of 401(k) plans on individual savings. We use the same dataset as in @CH2004 and @belloni2017program, which we refer to for more details and descriptive statistics. The data contain information about $n=9915$ observations from a sample of households from the 1991 Survey of Income and Program Participation (SIPP). We focus on the effect of eligibility for 401(k) plans ($D$) on total wealth ($Y$).[^17] We consider two different specifications of the control variables ($X$).

1.  **Two-way interactions specification.** We use the same set of controls as in @Benjamin2003 and @CH2004: Seven dummies for income categories, five dummies for age categories, family size, four dummies for education categories, as well as indicators of marital status, two-earner status, defined benefit pension status, individual retirement account (IRA) participation status, and homeownership. Following common empirical practice, we augment this baseline specification with all two-way interactions. After removing collinear columns there are $p=167$ control variables.

2.  **Quadratic spline & interactions specification.** This specification is due to @belloni2017program. It contains indicators of marital status, two-earner status, defined benefit pension status, IRA participation status, and homeownership status, second-order polynomials in family size and education, a third-order polynomial in age, a quadratic spline in income with six breakpoints, as well as interactions of all the non-income variables with each term in the income spline. After removing collinear columns there are $p=272$ control variables.

Table \[tab:results\_401k\] presents post double Lasso estimates based on the whole sample with $\lambda_{\text{BCCH}}$, $0.5\lambda_{\text{BCCH}}$ and $1.5\lambda_{\text{BCCH}}$. For comparison, we also report OLS estimates with and without covariates. For both specifications, the results are qualitatively similar across the different regularization choices and similar to OLS with all controls. This is not surprising given our simulation evidence, since $n$ is much larger than $p$. Nevertheless, there are some non-negligible quantitative differences between the point estimates. A comparison to OLS without control variables shows that omitting controls can yield substantial OVBs in this application.

[1.25]{}[1]{}

[lcc]{}\[tab:results\_401k\] & & [\
]{} [\
]{} Method & Point estimate & Robust std. error [\
]{} Post double Lasso ($\lambda_{\text{BCCH}}$) & 6624.47 & 2069.73\
Post double Lasso ($0.5\lambda_{\text{BCCH}}$) & 6432.36 & 2073.35\
Post double Lasso (1.5$\lambda_{\text{BCCH}}$) & 7474.51 & 2053.00\
OLS with all covariates & 6751.91 & 2067.86\
OLS without covariates & 35669.52 & 2412.02\
[\
]{} Method & Point estimate & Robust std. error [\
]{} Post double Lasso ($\lambda_{\text{BCCH}}$) & 4646.58 & 2014.63\
Post double Lasso ($0.5\lambda_{\text{BCCH}}$) & 5648.14 & 1988.81\
Post double Lasso (1.5$\lambda_{\text{BCCH}}$) & 4472.32 & 2027.72\
OLS with all covariates & 5988.41 & 2033.02\
OLS without covariates & 35669.52 & 2412.02\

[1.25]{}[1.4]{}

To investigate the impact of under-selection, we perform the following exercise. We draw random subsamples of size $n_s\in \{400,800,1600\}$ with replacement from the original dataset. This exercise mimics random sampling from a large super-population. Based on each subsample, we estimate $\alpha^\ast$ using post double Lasso with $\lambda_{\text{BCCH}}$, $0.5\lambda_{\text{BCCH}}$ and $1.5\lambda_{\text{BCCH}}$ and compute the bias as the difference between the average subsample estimate and the point estimate based on the original data with the same regularization parameters (cf. Table \[tab:results\_401k\]). The results are based on 2,000 simulation repetitions.

Figures \[fig:app\_401k\_1\] and \[fig:app\_401k\_2\] display the bias and the ratio of bias to standard deviation for both specifications. We find that post double Lasso can exhibit large finite sample biases. The biases under the quadratic spline & interactions specification tend to be smaller (in absolute value) than the biases under the two-way interactions specification. Interestingly, the ratio of bias to standard deviation may not be monotonically decreasing in $n_s$ (in absolute value) due to the standard deviation decaying faster than the bias. Finally, we find that post double Lasso can be very sensitive to the choice of the regularization parameters.

\[fig:app\_401k\_1\]

\[fig:app\_401k\_2\]

### Racial differences in the mental ability of children

In this section, we revisit @fryerlevitt2013’s analysis of the racial differences in the mental ability of young children. @fryerlevitt2013 use data from two sources: The Early Childhood Longitudinal Study Birth Cohort (ECLS-B) and the US Collaborative Perinatal Project (CPP). Here we focus on the CCP data, which contains information on women who gave birth in 12 medical centers from 1959 to 1965. We restrict the sample to black and white children and focus on the black-white test score gap. Our final sample includes $n=30002$ observations. We choose the standardized test score in the Stanford-Binet and Wechsler Intelligence Test at the age of seven as our outcome variable ($Y$).[^18] The variable of interest ($D$) is an indicator for black. Control variables ($X$) include rich information on socio-demographic characteristics, the home environment, and the prenatal environment. We refer to Table 1B in @fryerlevitt2013 for descriptive statistics. We use the same specification as in @fryerlevitt2013, excluding interviewer fixed effect.[^19] After removing collinear terms there are $p=78$ controls.

Table \[tab:results\_testscores\] shows the results for post double Lasso with $\lambda_{\text{BCCH}}$, $0.5\lambda_{\text{BCCH}}$ and $1.5\lambda_{\text{BCCH}}$, as well as OLS with and without covariates based on the whole sample. As expected, since $n=30002$ is much larger than $p=78$, all methods except for OLS without covariates yield similar results. [1.25]{}[1]{}

  \[tab:results\_testscores\]                                       
  ------------------------------------------------ ---------------- -------------------
  Method                                            Point estimate   Robust std. error
  Post double Lasso ($\lambda_{\text{BCCH}}$)          -0.6770            0.0114
  Post double Lasso ($0.5\lambda_{\text{BCCH}}$)       -0.6762            0.0114
  Post double Lasso (1.5$\lambda_{\text{BCCH}}$)       -0.6778            0.0114
  OLS with all covariates                              -0.6694            0.0115
  OLS without covariates                               -0.8538            0.0105

  : Results based on original data

[1.25]{}[1.4]{}

To investigate the impact of under-selection, we draw random subsamples of size $n_s\in \{400,800,1600\}$ with replacement from the original dataset. In each sample, we estimate $\alpha^\ast$ using post double Lasso with $\lambda_{\text{BCCH}}$, $0.5\lambda_{\text{BCCH}}$ and $1.5\lambda_{\text{BCCH}}$ and compute the bias as the difference between the average estimate based on the subsamples and the estimate based on the original data with the same regularization parameters. The results are based on 2,000 simulation repetitions.

Figure \[fig:app\_test\] displays the bias and the ratio of bias to standard deviation. While the bias is decreasing in $n_s$, it can be substantial and larger than the standard deviation when $n_s$ is small. Moreover, the performance of post double Lasso is very sensitive to the choice of the regularization parameters. For $\lambda=0.5\lambda_{\text{BCCH}}$, post double Lasso is approximately unbiased for all $n_s$, whereas, for $\lambda=1.5\lambda_{\text{BCCH}}$, the bias is comparable to the standard deviation even when $n_s=1600$.

\[fig:app\_test\]

Theoretical analysis {#sec:inference}
====================

This section provides a theoretical explanation for the findings in Section \[sec:evidence\]. We first establish a new necessary result for the Lasso’s inclusion and then derive lower and upper bounds on the OVBs of post double Lasso. These results hold for fixed $\left(n,\,p,\,k\right)$, and are also informative when $\frac{k\log p}{n}\rightarrow0$ or $\frac{k\log p}{n}\rightarrow\infty$. Throughout this section, we assume the regime where $p$ is comparable to or even much larger than $n$; that is, $p\asymp n$ or $p\gg n$.

For the convenience of the reader, here we collect the notation to be used in the theoretical analyses. Let $1_{m}$ denote the $m-$dimensional (column) vector of “1”s and $0_{m}$ is defined similarly. The $\ell_{1}-$norm of a vector $v\in\mathbb{R}^{m}$ is denoted by $\left|v\right|_{1}:=\sum_{i=1}^{m}\left|v_{i}\right|$. The $\ell_{\infty}$ matrix norm (maximum absolute row sum) of a matrix $A$ is denoted by $\left\Vert A\right\Vert _{\infty}:=\max_{i}\sum_{j}\left|a_{ij}\right|$. For a vector $v\in\mathbb{R}^{m}$ and a set of indices $T\subseteq\left\{ 1,\dots,m\right\} $, let $v_{T}$ denote the sub-vector (with indices in $T$) of $v$. For a matrix $A\in\mathbb{R}^{n\times m}$, let $A_{T}$ denote the submatrix consisting of the columns with indices in $T$. For a vector $v\in\mathbb{R}^{m}$, let $\textrm{sgn}(v):=\left\{ \textrm{sgn}(v_{j})\right\} _{j=1,...,m}$ denote the sign vector such that $\textrm{sgn}(v_{j})=1$ if $v_{j}>0$, $\textrm{sgn}(v_{j})=-1$ if $v_{j}<0$, and $\textrm{sgn}(v_{j})=0$ if $v_{j}=0$. We denote $\max\left\{ a,\,b\right\} $ by $a\vee b$ and $\min\left\{ a,\,b\right\} $ by $a\wedge b$.

Model setup
-----------

We consider the structural model , which can be written in matrix notation as $$\begin{aligned}
Y & = & D\alpha^{*}+X\beta^{*}+\eta,\label{eq:20}\\
D & = & X\gamma^{*}+v.\label{eq:21-1}\end{aligned}$$ Following standard practice, we work with centered data, i.e., $\bar{D}=\frac{1}{n}\sum_{i=1}^{n}D_{i}=0$, $\bar{X}=\left\{ \frac{1}{n}\sum_{i=1}^{n}X_{ij}\right\} _{j=1}^{p}=0_{p}$, and $\bar{Y}=\frac{1}{n}\sum_{i=1}^{n}Y_{i}=0$. In matrix notation, the reduced form becomes $$\begin{aligned}
Y & = & X\pi^{*}+u,\label{eq:reduce}\end{aligned}$$ where $\pi^{*}=\gamma^{*}\alpha^{*}+\beta^{*}$ and $u=\eta+\alpha^{*}v$. We make the following assumptions about model (\[eq:20\])(\[eq:21-1\]).

\[ass:inference\_normality\] The error terms $\eta$ and $v$ consist of independent entries drawn from $\mathcal{N}\left(0,\,\sigma_{\eta}^{2}\right)$ and $\mathcal{N}\left(0,\,\sigma_{v}^{2}\right)$, respectively, where $\eta$ and $v$ are independent of each other.

\[ass:inference\_sparsity\_incoherence\] The following are satisfied: (i) $\beta^{*}$ and $\gamma^{*}$ are exactly sparse with $k\left(\le\min\left\{ n,\,p\right\} \right)$ non-zero coefficients and $K=\left\{ j:\,\beta_{j}^{*}\neq0\right\} =\left\{ j:\,\gamma_{j}^{*}\neq0\right\} \neq\emptyset$; (ii) $$\left\Vert \left(X_{K^{c}}^{T}X_{K}\right)\left(X_{K}^{T}X_{K}\right)^{-1}\right\Vert _{\infty}=1-\phi\label{eq:14}$$ for some $\phi\in(0,\,1]$, where $K^{c}$ is the complement of $K$; (iii) $X_{K}^{T}X_{K}$ is a diagonal matrix with the diagonal entries $X_{j}^{T}X_{j}=s\neq0$ for all $j\in K$, and $X_{j}^{T}X_{j}\leq s$ for all $j\in K^{c}$.

Known as the incoherence condition due to @wainwright2009sharp, part (ii) in Assumption \[ass:inference\_sparsity\_incoherence\] is needed for the exclusion of the irrelevant controls. Note that if the columns in $X_{K^{c}}$ are orthogonal to the columns in $X_{K}$ (but within $X_{K^{c}}$, the columns need not be orthogonal to each other), then $\phi=1$. Obviously a special case of this is when the entire $X$ consists of mutually orthogonal columns (which is possible if $n\geq p$). To provide some intuition for (\[eq:14\]), let us consider the simple case where $k=1$ and $K=\left\{ 1\right\} $, $X$ is centered (such that $\left\{ \frac{1}{n}\sum_{i=1}^{n}X_{ij}\right\} _{j=1}^{p}=0_{p}$), and the columns in $X_{-1}=\left[X_{2},\,X_{3},\dots,\,X_{p}\right]$ are normalized such that the standard deviations of $X_{1}$ and $X_{j}$ (for any $j\in\left\{ 2,3,\dots,p\right\} $) are identical. Then, $1-\phi$ is simply the maximum of the absolute (sample) correlations between $X_{1}$ and each of the $X_{j}$s with $j\in\left\{ 2,3,\dots,p\right\} $.

Assumptions \[ass:inference\_normality\] and \[ass:inference\_sparsity\_incoherence\] are classical; in particular, parts (ii) and (iii) of Assumption \[ass:inference\_sparsity\_incoherence\] are often viewed favorable to the performance of Lasso. Our goal here is to demonstrate that, even in these simple yet classical settings, the finite sample OVBs of post double Lasso can be substantial relative to the standard deviation provided in the existing literature.

Stronger necessary results on the Lasso’s inclusion {#sec:nonasymptotic}
---------------------------------------------------

\[sec:main\_results\]

Post double Lasso exhibits OVBs whenever the relevant controls are selected in neither nor . To the best of our knowledge, there are no formal results strong enough to show that, with high probability, Lasso can fail to select the relevant controls in both steps. Therefore, we first establish a new necessary result for the (single) Lasso’s inclusion in Lemma \[prop:fixed\_design\]. Throughout Section \[sec:inference\], we focus on fixed designs (of $X$) to highlight the essence of the problem; see Appendix \[sec:Random-design\] for an extension to random designs.

\[prop:fixed\_design\] In model , suppose the $\varepsilon_{i}$s are independent over $i=1,\dots,n$ and $\varepsilon_{i}\sim\mathcal{N}\left(0,\sigma^{2}\right)$, where $\sigma\in\left(0,\,\infty\right)$;[^20] $\theta^{*}$ is exactly sparse with at most $k\left(\le\min\left\{ n,\,p\right\} \right)$ non-zero coefficients and $K=\left\{ j:\,\theta_{j}^{*}\neq0\right\} \neq\emptyset$. Let Assumption \[ass:inference\_sparsity\_incoherence\](ii)-(iii) hold. We solve the Lasso (\[eq:las\]) with $\lambda\geq\frac{2\sigma}{\phi}\sqrt{\frac{s}{n}}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}}$ (where $\tau>0$). Let $E_{1}$ denote the event that $\textrm{sgn}\left(\hat{\theta}_{j}\right)=-\textrm{sgn}\left(\theta_{j}^{*}\right)$ for at least one $j\in K$, and $E_{2}$ denote the event that $\textrm{sgn}\left(\hat{\theta}_{l}\right)=\textrm{sgn}\left(\theta_{l}^{*}\right)$ for at least one $l\in K$ with $$\left|\theta_{l}^{*}\right|\leq\frac{\lambda n}{2s}.\label{eq:min}$$ Then, we have $$\mathbb{P}\left(E_{1}\cap\mathcal{\mathcal{E}}\right)=\mathbb{P}\left(E_{2}\cap\mathcal{\mathcal{E}}\right)=0\label{eq:zero}$$ where $\mathcal{E}$ is defined in (\[eq:event\]) of Appendix \[appendix a1\] and $\mathbb{P}\left(\mathcal{E}\right)\geq1-\frac{1}{p^{\tau}}$.

If (\[eq:min\]) holds for all $l\in K$, we have $$\mathbb{P}\left(\hat{\theta}=0_{p}\right)\geq1-\frac{1}{p^{\tau}}.\label{eq:nec}$$

Lemma \[prop:fixed\_design\] shows that for large enough $p$, Lasso fails to select any of the relevant covariates with high probability if (\[eq:min\]) holds for all $l\in K$. If such conditions hold with respect to both and , then Lemma \[prop:fixed\_design\] implies that the relevant controls are selected in neither nor with high probability, i.e., at least $1-\frac{2}{p^{\tau}}$ (cf. Panels (a) and (b) of Figure \[fig:hist\_sel\_dml\] and Panel (a) of Figure \[fig:hist\_sel\_large\]).

Suppose that $$\lambda=\frac{2\sigma}{\phi}\sqrt{\frac{s}{n}}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}}.\label{eq:choice}$$ Then becomes $$\left|\theta_{l}^{*}\right|\sqrt{\frac{s}{n}}\leq\phi^{-1}\sigma\sqrt{\frac{2\left(1+\tau\right)\log p}{n}},\,\,\,\,l\in K.$$ The product “$\left|\theta_{l}^{*}\right|\sqrt{\frac{s}{n}}$” suggests that normalizing $X_{j}$ to make $\frac{1}{n}\sum_{i=1}^{n}X_{ij}^{2}=1$ for all $j=1,\dots,p$ does not change the conclusions in Lemma \[prop:fixed\_design\]. Such normalization simply leads to rescaled coefficients and estimates (by a factor of $\sqrt{\frac{s}{n}}$). In particular, the choice of $\lambda\geq\frac{2\sigma}{\phi}\sqrt{\frac{s}{n}}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}}$ in Lemma \[prop:fixed\_design\] becomes $\lambda=\lambda_{norm}\geq\frac{2\sigma}{\phi}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}}$; also, $\left|\theta_{j}^{*}\right|\leq\frac{\lambda n}{2s}$ (where $\lambda\geq\frac{2\sigma}{\phi}\sqrt{\frac{s}{n}}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}}$ without normalization) is replaced by $\sqrt{\frac{s}{n}}\left|\theta_{j}^{*}\right|\leq\frac{\lambda_{norm}}{2}$ (where $\lambda_{norm}\geq\frac{2\sigma}{\phi}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}}$ with normalization). For $n=10000$, $p=4000$, $\sigma=1$, $\phi=0.95$, and $\tau=0.5$ in (\[eq:choice\]), Lemma \[prop:fixed\_design\] says that, with probability at least $0.97$, none of the relevant covariates are selected if $\max_{l\in K}\left|\theta_{l}^{*}\right|\leq0.05$ for $\sqrt{\frac{s}{n}}=1$ and $\max_{l\in K}\left|\theta_{l}^{*}\right|\leq0.5$ for $\sqrt{\frac{s}{n}}=0.1$. Therefore, everything else equal, limited variability in the relevant controls makes it more likely for the Lasso to omit them. Limited variability in the irrelevant controls $X_{K^{c}}$ does not matter for the selection performance of the Lasso because, for $j\in K^{c}$, $\left|\theta_{j}^{*}\right|\sqrt{\frac{1}{n}X_{j}^{T}X_{j}}=0$ regardless.

Note that (\[eq:zero\]) implies $\mathbb{P}\left(\hat{\theta}_{l}\neq0\right)\leq\frac{1}{p^{\tau}}$ for any $l\in K$ subject to (\[eq:min\]). In comparison, @wainwright2009sharp shows that whenever $\theta_{l}^{*}\in\left(\lambda\frac{n}{s}\textrm{sgn}\left(\theta_{l}^{*}\right),\,0\right)$ or $\theta_{l}^{*}\in\left(0,\,\lambda\frac{n}{s}\textrm{sgn}\left(\theta_{l}^{*}\right)\right)$ for some $l\in K$, $$\mathbb{P}\left[\textrm{sgn}\left(\hat{\theta}_{K}\right)=\textrm{sgn}\left(\theta_{K}^{*}\right)\right]\leq\frac{1}{2}.\label{eq:wainwright}$$ Constant bounds in the form of (\[eq:wainwright\]) cannot explain that, with high probability, Lasso fails to select the relevant covariates in both and when $p$ is sufficiently large.

\[more choices\]Under the assumptions in Lemma \[prop:fixed\_design\], the choices of regularization parameters $\lambda_{1}$ and $\lambda_{2}$ coincide with those in @bickel2009simultaneous when $\phi=1$; e.g., the columns in $X_{K^{c}}$ are orthogonal to the columns in $X_{K}$ (but within $X_{K^{c}}$, the columns need not be orthogonal to each other). If $\phi\approx 1$, similar results as those in Lemma \[prop:fixed\_design\] as well as in Sections \[sec:bias\_post\_double\_lasso\], \[sec:post\_double\_upper\_bounds\], and \[sec:bias\_debiased\_lasso\] hold for any choices of regularization parameters derived from the principle that $\lambda$ should be no smaller than $2\max_{j=1,\dots,p}\left|\frac{X_{j}^{T}\varepsilon}{n}\right|$ with high probability. These choices constitute what has been used in the vast majority of literature [e.g., @bickel2009simultaneous; @wainwright2009sharp; @belloni2012sparse; @belloni2013least; @belloni2014inference].

Choosing regularization parameters according to the principle discussed in Remark \[more choices\] prevents the inclusion of overly many irrelevant controls. This property is needed to ensure a good performance of post double Lasso, as discussed in Section \[sec:simulation\_evidence\] and Footnote \[footnote:over-selection\].

Lower bounds on the OVBs {#sec:bias_post_double_lasso}
------------------------

Proposition \[prop:bias\_post\_double\_formula\] derives a lower bound formula for the OVB of post double Lasso. We focus on the case where $\alpha^{*}=0$ because the conditions required to derive the explicit formula are difficult to interpret when $\alpha^{*}\ne0$. The reason is that the error in the reduced form equation involves $\alpha^{*}$, such that the choice of $\lambda_{1}$ in depends on the unknown $\alpha^{*}$. On the other hand, it is possible to provide easy-to-interpret scaling results (without explicit constants) for cases where $\alpha^{*}\ne0$, as we will show in Propositions \[prop:bias\_post\_double\_selection\] and \[prop:bias\_post\_double\_selection-1\].

\[prop:bias\_post\_double\_formula\] Let Assumptions \[ass:inference\_normality\] and \[ass:inference\_sparsity\_incoherence\] hold. Suppose $\lambda_{1}=2\phi^{-1}\sigma_{\eta}\sqrt{\frac{s}{n}}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}}$, $\lambda_{2}=2\phi^{-1}\sigma_{v}\sqrt{\frac{s}{n}}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}}$; for all $j\in K$ and $\left|a\right|,\left|b\right|\in(0,\,1]$, $$\text{both}\quad\beta_{j}^{*}\sqrt{\frac{s}{n}}=a\phi^{-1}\sigma_{\eta}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}}\quad\text{ and}\quad\gamma_{j}^{*}\sqrt{\frac{s}{n}}=b\phi^{-1}\sigma_{v}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}}.\label{eq:16}$$ In terms of $\tilde{\alpha}$ obtained from (\[eq:double\]), we have $$\left|\mathbb{E}\left(\tilde{\alpha}-\alpha^{*}\vert\mathcal{M}\right)\right|\geq\underset{:=\underline{\text{OVB}}}{\underbrace{\max_{r\in(0,1]}T_{1}\left(r\right)T_{2}\left(r\right)}}$$ where $$\begin{aligned}
        T_{1}\left(r\right) & = & \frac{\left(1+\tau\right)\left|ab\right|\phi^{-2}\sigma_{\eta}\frac{k\log p}{n}}{4\left(1+\tau\right)\phi^{-2}b^{2}\sigma_{v}\frac{k\log p}{n}+\left(1+r\right)\sigma_{v}},\\
        T_{2}\left(r\right) & = & 1-k\exp\left(\frac{-b^{2}\left(1+\tau\right)\log p}{4\phi^{2}}\right)-\frac{1}{p^{\tau}}-\exp\left(\frac{-nr^{2}}{8}\right),
    \end{aligned}$$ for any $r\in(0,\,1]$, and $\mathcal{M}$ is an event with $\mathbb{P}\left(\mathcal{M}\right)\geq1-k\exp\left(\frac{-b^{2}\left(1+\tau\right)\log p}{4\phi^{2}}\right)-\frac{2}{p^{\tau}}$.[^21]

\[rem:absenceOVB\] Let Assumptions \[ass:inference\_normality\] and \[ass:inference\_sparsity\_incoherence\] hold. As in Proposition \[prop:bias\_post\_double\_formula\], let $\lambda_{1}=2\phi^{-1}\sigma_{\eta}\sqrt{\frac{s}{n}}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}}$ and $\lambda_{2}=2\phi^{-1}\sigma_{v}\sqrt{\frac{s}{n}}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}}$. For all $j\in K,\;a>3,\,b>3$, if $$\text{either}\quad\left|\beta_{j}^{*}\right|\sqrt{\frac{s}{n}}=a\phi^{-1}\sigma_{\eta}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}}\quad\text{or}\quad\left|\gamma_{j}^{*}\right|\sqrt{\frac{s}{n}}=b\phi^{-1}\sigma_{v}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}},\label{eq:15}$$ then $\mathbb{P}\left[\textrm{supp}\left(\hat{\pi}\right)=\textrm{supp}\left(\pi^{*}\right)\right]\geq1-\frac{1}{p^{\tau}}$ or $\mathbb{P}\left[\textrm{supp}\left(\hat{\gamma}\right)=\textrm{supp}\left(\gamma^{*}\right)\right]\geq1-\frac{1}{p^{\tau}}$ by standard arguments. As a result, $\mathbb{P}\left(\left\{ \hat{I}_{1}\cup\hat{I}_{2}\right\} =K\right)\geq1-\frac{1}{p^{\tau}}$, where $\hat{I}_{1}$ and $\hat{I}_{2}$ are defined in (\[eq:double\]); i.e., the final OLS step (\[eq:double\]) includes all the relevant controls with high probability. By similar argument as in Section \[sec:appendix\_post\_double\_lasso\], on the high probability event $\left\{ \left\{ \hat{I}_{1}\cup\hat{I}_{2}\right\} =K\right\} $, the OVB of $\tilde{\alpha}$ is zero. Panels (d) of Figures \[fig:fsd\_dml\]\[fig:hist\_sel\_dml\] and Panels (b) of Figures \[fig:fsd\_large\]\[fig:hist\_sel\_large\] illustrate this phenomenon.

Proposition \[prop:bias\_post\_double\_formula\] and Remark \[rem:absenceOVB\] show that the OVBs can be drastically different when we fix $\left(n,\,p,\,k,\,\sigma_{v},\,\sigma_{\eta},\,\alpha^{*}\right)$ in (\[eq:20\])(\[eq:21-1\]), but vary the products of the absolute values of the non-zero coefficients and the standard deviations of the corresponding controls:

-   Under (\[eq:16\]), $\mathbb{P}\left(\hat{I}_{1}=\hat{I}_{2}=\emptyset\right)\geq1-\frac{2}{p^{\tau}}$ and the OVBs are bounded from below by $\underline{OVB}$, which decreases as $\left|a\right|$ decreases. By fixing $\left(n,p,\phi,\sigma_{\eta},\sigma_{v},\tau\right)$, one can easily see from (\[eq:16\]) that the lower bound $\underline{OVB}$ depends on $\left|\beta_{j}^{*}\right|\sqrt{\frac{s}{n}}$ through $\left|a\right|$ and $\left|\gamma_{j}^{*}\right|\sqrt{\frac{s}{n}}$ through $\left|b\right|$, for $j\in K$. Therefore, under-selection cannot be avoided or mitigated by rescaling the covariates and $\underline{OVB}$ remains the same after rescaling. This fact suggests that the problem of “limited‘" variability can be recast as a “small‘" coefficient problem and vice versa.

-   Under (\[eq:15\]), $\mathbb{P}\left(\left\{ \hat{I}_{1}\cup\hat{I}_{2}\right\} =K\right)\geq1-\frac{1}{p^{\tau}}$ and the OVBs are zero.

Under Assumptions \[ass:inference\_normality\]–\[ass:inference\_sparsity\_incoherence\] and if $\sigma_{v}$ is bounded away from zero and $\sigma_{\eta}$ is bounded from above, by contrast, the existing theory would imply that the biases of post double Lasso are bounded from above by $\texttt{constant}\cdot\frac{k\log p}{n}$, irrespective of whether Lasso fails to select the relevant controls or not, and how small $\left|a\right|$ and $\left|b\right|$ are. The (positive) $\texttt{constant}$ here does not depend on $\left(n,\,p,\,k,\,\frac{s}{n},\,\beta_{K}^{*},\,\gamma_{K}^{*},\,\alpha^{*}\right)$, and bears little meaning in the asymptotic framework which simply assumes $\frac{k\log p}{\sqrt{n}}\rightarrow0$ among other sufficient conditions. \[The existing theoretical framework makes it difficult to derive an informative $\texttt{constant}$, and to our knowledge, the literature provides no such derivation.\] The asymptotic upper bound $\texttt{constant}\cdot\frac{k\log p}{n}$ corresponds to the least favorable case and, thus, is uninformative about the most favorable cases that could vary in $\left(\frac{s}{n},\,\beta_{K}^{*},\,\gamma_{K}^{*},\,\alpha^{*}\right)$. By contrast, our lower bound analyses are informative about the most favorable cases, which are crucial for understanding the finite sample limitations of post double Lasso.

For sufficiently large $p$, the configuration of $\left|\beta_{K}^{*}\right|$, $\left|\gamma_{K}^{*}\right|$, and $\frac{s}{n}$ in Proposition \[prop:bias\_post\_double\_formula\] leads to large $\mathbb{P}\left(\hat{I}_{1}=\hat{I}_{2}=\emptyset\right)$; meanwhile, the lower bound $\underline{OVB}$ is characterized by the interplay between $\mathbb{P}\left(\hat{I}_{1}=\hat{I}_{2}=\emptyset\right)$ (related to $T_{2}$) and the omitted coefficients (related to $T_{1}$). To gauge the magnitude of the OVB and explain why the confidence intervals proposed in the literature can exhibit under-coverage, it is instructive to compare $\underline{OVB}$ with $\ensuremath{\sigma_{\tilde{\alpha}}=\frac{1}{\sqrt{n}}\frac{\sigma_{\eta}}{\sigma_{v}}}$, the standard deviation (of $\tilde{\alpha}$) obtained from the asymptotic distribution in @belloni2014inference.[^22] Let us consider the following examples: $a=b=1$, $\sigma_{\eta}=\sigma_{v}=1$, $n=10000$, $p=4000$, $\tau=0.5$, and $\phi=0.95$; if $\left|\beta_{j}^{*}\right|\sqrt{\frac{s}{n}}\approx0.05$ and $\left|\gamma_{j}^{*}\right|\sqrt{\frac{s}{n}}\approx0.05$ for all $j\in K$, then $\frac{\underline{OVB}}{\sigma_{\tilde{\alpha}}}\approx0.52$ for $k=5$ and $\frac{\underline{OVB}}{\sigma_{\tilde{\alpha}}}\approx0.12$ for $k=1$. It is important to bear in mind that, the calculations of $\frac{\underline{OVB}}{\sigma_{\tilde{\alpha}}}$ are based on our theoretical *lower* bounds (for the OVBs) corresponding to the most favorable cases. The result $\frac{\underline{OVB}}{\sigma_{\tilde{\alpha}}}\approx0.52$ equals to the ratio of $\texttt{constant}\cdot\frac{k\log p}{n}$ (the typical asymptotic *upper* bound for the bias) to $\frac{1}{\sqrt{n}}$ (the typical scaling of the standard deviation $\sigma_{\tilde{\alpha}}$), with $\texttt{constant}=1.25$; for the second example with $k=1$, $\frac{\underline{OVB}}{\sigma_{\tilde{\alpha}}}\approx1.47\frac{k\log p}{\sqrt{n}}$. This comparison suggests that these *most favorable* cases are essentially the *least favorable* case. Now, let us recall the first example where $k=5$; changing $a=1$ to $a=0.1$ there yields $\frac{\underline{OVB}}{\sigma_{\tilde{\alpha}}}\approx0.12\frac{k\log p}{\sqrt{n}}$ when $\left|\beta_{j}^{*}\right|\sqrt{\frac{s}{n}}\approx0.005$, while changing $a=1$ to $a=3.1$ yields no OVBs when $\left|\beta_{j}^{*}\right|\sqrt{\frac{s}{n}}\approx0.16$, according to Remark \[rem:absenceOVB\]. These examples suggest that the most favorable cases may differ substantially from the least favorable case.

In view of Proposition \[prop:bias\_post\_double\_formula\], $\underline{OVB}$ is not a simple linear function of $\frac{k\log p}{n}$ in general, but rather depends on $\left(n,p,k\right)$ and other factors in a more complex way. As $\frac{k\log p}{n}\rightarrow\infty$, we will show in Propositions \[prop:bias\_post\_double\_selection\] and \[prop:bias\_post\_double\_selection-1\] that, the OVB lower bounds scale as $\left|\alpha^{*}\right|$ or as $\frac{\sigma_{\eta}}{\sigma_{v}}$, depending on the configuration of $\left(\beta_{K}^{*},\,\gamma_{K}^{*},\,\alpha^{*}\right)$; as a consequence, $\frac{\underline{OVB}}{\sigma_{\tilde{\alpha}}}$ scales as $\frac{\sqrt{n}\left|\alpha^{*}\right|\sigma_{v}}{\sigma_{\eta}}$ or as $\sqrt{n}$. While the scaling $\sqrt{n}$ as $\frac{k\log p}{n}\rightarrow\infty$ can be easily seen from Proposition \[prop:bias\_post\_double\_formula\] where $\alpha^{*}=0$, Propositions \[prop:bias\_post\_double\_selection\] and \[prop:bias\_post\_double\_selection-1\] also consider cases where $\alpha^{*}\neq0$. When double under-selection occurs with high probability and $\frac{k\log p}{\sqrt{n}}$ is not small enough, according to the results above, $\tilde{\alpha}$ will perform poorly in general, except in one (albeit) extreme case. By Lemma \[prop:fixed\_design\] and (\[eq:16\]), $\left|\hat{\beta}-\beta^{*}\right|_{1}=\left|a\right|\frac{k\lambda_{1}n}{2s}$ and $\left|\hat{\gamma}-\gamma^{*}\right|_{1}=\left|b\right|\frac{k\lambda_{2}n}{2s}$ with probability at least $1-\frac{2}{p^{\tau}}$. If $\sigma_{v}$ is bounded away from zero, $\sigma_{\eta}$ is bounded from above, and $$\left|a\right|=\left|b\right|=o\left(1\right),\label{eq: small_OVB}$$ by similar argument in @belloni2014inference, we can show that $\sqrt{n}\left(\tilde{\alpha}-\alpha^{*}\right)$ is approximately normal and unbiased, *even if $\frac{k\log p}{\sqrt{n}}$ scales as a constant*. *If $\frac{k\log p}{n}$ scales as a constant*, then replacing (\[eq: small\_OVB\]) with $$\left|a\right|=\left|b\right|=o\left(\frac{1}{\sqrt{n}}\right),\label{eq: smaller_OVB}$$ yields the same conclusion. Holding other factors constant, the magnitude of OVBs decreases as $\left|\beta_{K}^{*}\right|$ and $\left|\gamma_{K}^{*}\right|$ decrease (i.e., as $\left|a\right|$ and $\left|b\right|$ decrease). As $\left|a\right|$ and $\left|b\right|$ become very small, the relevant controls become essentially irrelevant.

The next results, Propositions \[prop:bias\_post\_double\_selection\] and \[prop:bias\_post\_double\_selection-1\], provide the scaling of OVB lower bounds under two different setups. These results consider cases where $\alpha^{*}\neq0$. Depending on the setups, some of the cases behave similarly to the case where $\alpha^{*}=0$ and some behave differently. For functions $f(n)$ and $g(n)$, we write $f(n)\succsim g(n)$ to mean that $f(n)\geq cg(n)$ for a universal constant $c\in(0,\,\infty)$ and similarly, $f(n)\precsim g(n)$ to mean that $f(n)\leq c^{'}g(n)$ for a universal constant $c^{'}\in(0,\,\infty)$; $f(n)\asymp g(n)$ when $f(n)\succsim g(n)$ and $f(n)\precsim g(n)$ hold simultaneously. As a general rule, $c$ constants denote positive universal constants that are independent of $n$, $p$, $k$, $\sigma_{\eta}$, $\sigma_{v}$, $s$, $\alpha^{*}$ and may change from place to place.

These propositions are useful for understanding how the OVB lower bounds behave roughly as a function of $\left(n,\,p,\,k,\,\sigma_{\eta},\,\sigma_{v},\,\left|\alpha^{*}\right|\right)$, by abstracting from Proposition \[prop:bias\_post\_double\_formula\] all the finite sample subtleties (such as various universal constants) and assuming $\left|a\right|,\,\left|b\right|\asymp1$ (while $\left|a\right|,\,\left|b\right|\le1$) and $\phi^{-1}\precsim1$ (hence, $\phi^{-1}\asymp1$). Propositions \[prop:bias\_post\_double\_selection\] and \[prop:bias\_post\_double\_selection-1\] suggest that the OVB lower bounds do not simply scale as $\frac{k\log p}{n}$ even if $\left|a\right|,\,\left|b\right|\asymp1$ and $\phi^{-1}\precsim1$: When $\frac{k\log p}{n}\rightarrow0$, they scale as $\frac{\sigma_{\eta}}{\sigma_{v}}\frac{k\log p}{n}$ (in Proposition \[prop:bias\_post\_double\_selection\]) and as $\left|\alpha^{*}\right|\frac{k\log p}{n}$ (in Proposition \[prop:bias\_post\_double\_selection-1\]); when $\frac{k\log p}{n}\rightarrow\infty$, they scale as $\frac{\sigma_{\eta}}{\sigma_{v}}$ (in Proposition \[prop:bias\_post\_double\_selection\]) and as $\left|\alpha^{*}\right|$ (in Proposition \[prop:bias\_post\_double\_selection-1\]), instead of $\infty$.

\[prop:bias\_post\_double\_selection\] Let Assumptions \[ass:inference\_normality\] and \[ass:inference\_sparsity\_incoherence\] hold. Suppose $\phi^{-1}\precsim1$ in (\[eq:14\]); the regularization parameters in (\[eq:las-1\]) and (\[eq:las-2\]) are chosen in a similar fashion as in Lemma \[prop:fixed\_design\] such that $\lambda_{1}\asymp\phi^{-1}\sigma_{\eta}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}$[^23] and $\lambda_{2}\asymp\phi^{-1}\sigma_{v}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}$; for all $j\in K$, $\left|\beta_{j}^{*}\right|\leq\frac{\lambda_{1}n}{2s}$ and $\left|\gamma_{j}^{*}\right|\leq\frac{\lambda_{2}n}{2s}$, but $\left|\beta_{j}^{*}\right|\asymp\sigma_{\eta}\sqrt{\frac{n}{s}}\sqrt{\frac{\log p}{n}}$ and $\left|\gamma_{j}^{*}\right|\asymp\sigma_{v}\sqrt{\frac{n}{s}}\sqrt{\frac{\log p}{n}}$. Let us consider $\tilde{\alpha}$ obtained from (\[eq:double\]).

\(i) If $\alpha^{*}=0$, then there exist positive universal constants $c^{\dagger},c_{1},c_{2},c_{3},c^{*},c_{0}^{*}$ such that $$\left|\mathbb{E}\left(\tilde{\alpha}-\alpha^{*}\vert\mathcal{M}\right)\right|\geq c^{\dagger}\frac{\sigma_{\eta}}{\sigma_{v}}\left(\frac{k\log p}{n}\wedge1\right)\left[1-c_{1}k\exp\left(-c_{2}\log p\right)-\exp\left(-c_{3}n\right)\right],\label{eq:22}$$ where $\mathcal{M}$ is an event with $\mathbb{P}\left(\mathcal{M}\right)\geq1-c^{*}k\exp\left(-c_{0}^{*}\log p\right)$.

\(ii) If $\alpha^{*}\gamma_{j}^{*}\in(0,\,-\beta_{j}^{*}]$, $\beta_{j}^{*}<0$ for $j\in K$ (or, $\alpha^{*}\gamma_{j}^{*}\in[-\beta_{j}^{*},\,0)$, $\beta_{j}^{*}>0$ for $j\in K$), then for some positive universal constants $c^{\dagger},c_{1},c_{2},c_{3},c^{*},c_{0}^{*}$, (\[eq:22\]) holds with $\mathbb{P}\left(\mathcal{M}\right)\geq1-c^{*}k\exp\left(-c_{0}^{*}\log p\right)$.

To motivate the next proposition, note that as long as $\hat{I}_{1}=K$ or $\hat{I}_{2}=K$, the final OLS step (\[eq:double\]) corresponds to the oracle estimator. Under sufficient variability in $X_{K}$, when $\left|\pi_{K}^{*}\right|$ are small enough (so $\hat{I}_{1}=\emptyset$ with high probability), but $\left|\gamma_{K}^{*}\right|$ in (\[eq:21-1\]) are large enough (so $\hat{I}_{2}=K$ with high probability), post double Lasso coincides with the oracle estimator while OLS post (the single) Lasso (\[eq:las-1\]) omits the relevant controls with high probability. As we will show below, if $\left|\gamma_{K}^{*}\right|$ are also small enough, then like the OLS post (\[eq:las-1\]), post double Lasso can also yield substantial OVBs relative to the standard deviation.

\[prop:bias\_post\_double\_selection-1\] Let Assumptions \[ass:inference\_normality\] and \[ass:inference\_sparsity\_incoherence\] hold. Suppose $\phi^{-1}\precsim1$ in (\[eq:14\]); the regularization parameters in (\[eq:las-1\]) and (\[eq:las-2\]) are chosen in a similar fashion as in Lemma \[prop:fixed\_design\] such that $\lambda_{1}\asymp\frac{\left(\sigma_{\eta}+\left|\alpha^{*}\right|\sigma_{v}\right)}{\phi}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}$ and $\lambda_{2}\asymp\phi^{-1}\sigma_{v}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}$; for all $j\in K$, $\left|\gamma_{j}^{*}\right|\leq\frac{\lambda_{2}n}{2s}$ but $\left|\gamma_{j}^{*}\right|\asymp\sigma_{v}\sqrt{\frac{n}{s}}\sqrt{\frac{\log p}{n}}$. Let us consider $\tilde{\alpha}$ obtained from (\[eq:double\]).

\(i) If $\pi_{j}^{*}=0$ for all $j\in K$, then there exist positive universal constants $c^{\dagger},c_{1},c_{2},c_{3},c^{*},c_{0}^{*}$ such that $$\left|\mathbb{E}\left(\tilde{\alpha}-\alpha^{*}\vert\mathcal{M}\right)\right|\geq c^{\dagger}\left|\alpha^{*}\right|\left(\frac{k\log p}{n}\wedge1\right)\left[1-c_{1}k\exp\left(-c_{2}\log p\right)-\exp\left(-c_{3}n\right)\right],\label{eq:23}$$ where $\mathbb{P}\left(\mathcal{M}\right)\geq1-c^{*}k\exp\left(-c_{0}^{*}\log p\right)$.

\(ii) For all $j\in K$, suppose $\left|\pi_{j}^{*}\right|\asymp\left(\sigma_{\eta}+\left|\alpha^{*}\right|\sigma_{v}\right)\sqrt{\frac{n}{s}}\sqrt{\frac{\log p}{n}}$, and we have either (1) $\alpha^{*}<0$, $\beta_{j}^{*}>0$, $\gamma_{j}^{*}>0$, $0<\pi_{j}^{*}\leq\frac{\lambda_{1}n}{2s}$, or (2) $\alpha^{*}>0$, $\beta_{j}^{*}<0$, $\gamma_{j}^{*}>0$, $-\frac{\lambda_{1}n}{2s}<\pi_{j}^{*}<0$. Then there exist positive universal constants $c^{\dagger},c_{1},c_{2},c_{3},c^{*},c_{0}^{*}$ such that (\[eq:23\]) holds with $\mathbb{P}\left(\mathcal{M}\right)\geq1-c^{*}k\exp\left(-c_{0}^{*}\log p\right)$.

The different scaling in Propositions \[prop:bias\_post\_double\_selection\] and \[prop:bias\_post\_double\_selection-1\] comes from the fact that, the former constrains the magnitude of $\pi_{K}^{*}$ through constraining the magnitude of $\left(\beta_{K}^{*},\,\gamma_{K}^{*},\,\alpha^{*}\right)$, and the latter allows large magnitude of $\left(\beta_{K}^{*},\,\alpha^{*}\right)$ while constraining the magnitude of $\left(\pi_{K}^{*},\,\gamma_{K}^{*}\right)$. Proposition \[prop:bias\_post\_double\_selection-1\] suggests that, even if $\frac{k\log p}{\sqrt{n}}$ is small, the OVB lower bounds can still be substantial if $\left|\alpha^{*}\right|$ is large enough. To see this, suppose $\left|\alpha^{*}\right|\asymp\frac{n}{k\log p}$ and $\frac{k\log p}{n}=o\left(\frac{1}{\sqrt{n}}\right)$ (so that $\frac{k\log p}{\sqrt{n}}=o\left(1\right)$). Then the OVB lower bounds in Proposition \[prop:bias\_post\_double\_selection-1\] scale as a constant bounded away from zero. Even if $\frac{\sigma_{\eta}}{\sigma_{v}}\asymp1$, the ratios of these lower bounds to $\ensuremath{\sigma_{\tilde{\alpha}}=\frac{1}{\sqrt{n}}\frac{\sigma_{\eta}}{\sigma_{v}}}$ scale as $\frac{\left|\alpha^{*}\right|\sigma_{v}}{\sigma_{\eta}}\frac{k\log p}{\sqrt{n}}\asymp\sqrt{n}$. By contrast, in the setup of Proposition \[prop:bias\_post\_double\_selection\], the ratios of its lower bounds to $\ensuremath{\sigma_{\tilde{\alpha}}}$ scale as $\frac{k\log p}{\sqrt{n}}=o\left(1\right)$ (irrespective of $\frac{\sigma_{\eta}}{\sigma_{v}}$), whenever $\frac{k\log p}{n}=o\left(\frac{1}{\sqrt{n}}\right)$.

Propositions \[prop:bias\_post\_double\_formula\]\[prop:bias\_post\_double\_selection-1\] on the OVB lower bounds have important implications for inference based on the post double Lasso procedure proposed in @belloni2014inference. On the one hand, the existing theoretical results on post double Lasso such as @belloni2014inference prove that the assumptions $\frac{k\log p}{\sqrt{n}}=o\left(1\right)$, $\left|\alpha^{*}\right|\precsim 1$, $\left|\sigma_{v}\right|\asymp 1$, and $\left|\sigma_{\eta}\right|\asymp 1$ (among other regularity conditions) are *sufficient* for establishing asymptotic normality and unbiasedness, regardless of whether under-selection is present or not. On the other hand, our Propositions \[prop:bias\_post\_double\_formula\]\[prop:bias\_post\_double\_selection-1\] show settings where double under-selection occurs with high probability, and suggest that in such settings, small $\frac{k\log p}{\sqrt{n}}$ and $\frac{\left|\alpha^{*}\right|\sigma_{v}}{\sigma_{\eta}}$ are *almost necessary* for a reliable performance of the post double Lasso, under any regularization choice derived from the principle mentioned in Remark \[more choices\].

\[rem:when\_post\_double\_work\] In view of our theoretical results and simulation evidence, if increasing $\lambda_{1}$ and $\lambda_{2}$ from the theoretically recommended choices (such as the one in @belloni2014inference) yield similar $\tilde{\alpha}$s, then the underlying model could be in the regime where either under-selection in both Lasso steps is unlikely, or the OVBs are simply negligible. Outside this regime, the magnitude and performance of post double Lasso can be quite sensitive to the increase of $\lambda_{1}$ and $\lambda_{2}$. The rationale behind this heuristic lies in that the final step of post double Lasso, (\[eq:double\]), is simply an OLS regression of $Y$ on $D$ and the union of selected controls from (\[eq:las-1\])(\[eq:las-2\]). One might ask by how much $\lambda_{1}$ and $\lambda_{2}$ should be increased for the robustness checks. For the regularization choice proposed in @belloni2014inference, our simulations suggest that an increase by $50\%$ works well in practice. As an example, Figure \[fig:dml\_bias\_std\_sens\] shows that the ratio of bias to standard deviation for the post double Lasso is less sensitive to increasing the regularization parameters when $\sigma_{x}\ge0.5$ because the selection performance of Lasso is more robust when $\sigma_{x}\ge 0.5$ (cf. Figures \[fig:sel\_sensitivity\] and \[fig:sel\_rel\_sensitivity\]). [\[]{}In Figures \[fig:sel\_sensitivity\], \[fig:sel\_rel\_sensitivity\], and \[fig:dml\_bias\_std\_sens\], we fix $\beta_{K}^{*}$ and $\gamma_{K}^{*}$ while varying $\sigma_{x}$, which is equivalent to varying the magnitude of $\beta_{K}^{*}$ and $\gamma_{K}^{*}$ while fixing $\sigma_{x}$, as discussed previously. In other words, we can simply replace the label $\sigma_{x}$ for the horizontal axis by the magnitude of $\beta_{K}^{*}$ and $\gamma_{K}^{*}$.[\]]{}

Upper bounds on the OVBs {#sec:post_double_upper_bounds}
------------------------

Proposition \[prop:bias\_post\_double\_selection\] suggests that, the lower bounds on the OVBs scale as $\frac{\sigma_{\eta}}{\sigma_{v}}$ as $\frac{k\log p}{n}\rightarrow\infty$. Proposition \[prop:bias\_post\_double\_selection-1\] suggests that, the lower bounds on the OVBs scale as $\left|\alpha^{*}\right|$ as $\frac{k\log p}{n}\rightarrow\infty$. This prompts the question of how large the OVBs can be under the regime where $\frac{k\log p}{n}\rightarrow\infty$. To answer this question, we would like a meaningful upper bound (other than just $\infty$) on the OVBs, but there is no such result in the existing literature. Interestingly enough, we show that the upper bounds on the OVBs scale as $\frac{\sigma_{\eta}}{\sigma_{v}}$ or $\left|\alpha^{*}\right|$ with high probability, even if $\frac{k\log p}{n}\rightarrow\infty$ and the Lasso is inconsistent in the sense $\sqrt{\frac{1}{n}\sum_{i=1}^{n}\left(X_{i}\hat{\pi}-X_{i}\pi^{*}\right)^{2}}\rightarrow\infty$, $\sqrt{\frac{1}{n}\sum_{i=1}^{n}\left(X_{i}\hat{\gamma}-X_{i}\gamma^{*}\right)^{2}}\rightarrow\infty$ with high probability. To illustrate how the OVB upper bounds behave roughly as a function of $\left(n,\,p,\,k,\,\sigma_{\eta},\,\sigma_{v},\,\left|\alpha^{*}\right|\right)$, we again abstract from all the finite sample subtleties (such as various universal constants) and assume $\left|a\right|,\,\left|b\right|\asymp1$ (while $\left|a\right|,\,\left|b\right|\le1$) and $\phi^{-1}\precsim1$.

\[prop:bias\_post\_double\_selection-upper\]Let Assumptions \[ass:inference\_normality\] and \[ass:inference\_sparsity\_incoherence\] hold. Suppose $\phi^{-1}\precsim1$ in (\[eq:14\]); the regularization parameters in (\[eq:las-1\]) and (\[eq:las-2\]) are chosen in a similar fashion as in Proposition \[prop:bias\_post\_double\_selection\] such that $\lambda_{1}\asymp\phi^{-1}\sigma_{\eta}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}$ and $\lambda_{2}\asymp\phi^{-1}\sigma_{v}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}$; for all $j\in K$, $\gamma_{j}^{*}=\gamma^{*}$, $\left|\beta_{j}^{*}\right|\leq\frac{\lambda_{1}n}{2s}$ and $\left|\gamma^{*}\right|\leq\frac{\lambda_{2}n}{2s}$, but $\left|\beta_{j}^{*}\right|\asymp\sigma_{\eta}\sqrt{\frac{n}{s}}\sqrt{\frac{\log p}{n}}$ and $\left|\gamma^{*}\right|\asymp\sigma_{v}\sqrt{\frac{n}{s}}\sqrt{\frac{\log p}{n}}$. Let us consider $\tilde{\alpha}$ obtained from (\[eq:double\]). Then for either $\alpha^{*}=0$, or $\alpha^{*}\neq0$ subject to the conditions in part (ii) of Proposition \[prop:bias\_post\_double\_selection\], there exist positive universal constants $c_{1},c_{2},c_{3},c_{4},c^{*},c_{0}^{*}$ such that $$\mathbb{P}\left(\left|\tilde{\alpha}-\alpha^{*}\right|\leq\overline{OVB}\vert\hat{I}_{1}=\hat{I}_{2}=\emptyset\right)\geq1-c_{1}k\exp\left(-c_{2}\log p\right)-c_{3}\exp\left(-c_{4}nr^{2}\right)$$ where $\mathbb{P}\left(\hat{I}_{1}=\hat{I}_{2}=\emptyset\right)\geq1-c^{*}\exp\left(-c_{0}^{*}\log p\right)$ and $$\overline{OVB}\asymp\max\left\{ \frac{\sigma_{\eta}}{\sigma_{v}}\left(\frac{k\log p}{n}\wedge1\right),\,\frac{\sigma_{v}\sigma_{\eta}\left(r\vee\frac{k\log p}{n}\right)}{\left(\frac{k\log p}{n}\vee1\right)\sigma_{v}^{2}}\right\}$$ for any $r\in(0,\,1]$.

\[prop:bias\_post\_double\_selection-upper-1\]Let Assumptions \[ass:inference\_normality\] and \[ass:inference\_sparsity\_incoherence\] hold. Suppose $\phi^{-1}\precsim1$ in (\[eq:14\]); the regularization parameters in (\[eq:las-1\]) and (\[eq:las-2\]) are chosen in a similar fashion as in Proposition \[prop:bias\_post\_double\_selection-1\] such that $\lambda_{1}\asymp\frac{\sigma_{\eta}+\left|\alpha^{*}\right|\sigma_{v}}{\phi}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}$ and $\lambda_{2}\asymp\phi^{-1}\sigma_{v}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}$; for all $j\in K$, $\gamma_{j}^{*}=\gamma^{*}$, $\left|\gamma^{*}\right|\leq\frac{\lambda_{2}n}{2s}$, but $\left|\gamma^{*}\right|\asymp\sigma_{v}\sqrt{\frac{n}{s}}\sqrt{\frac{\log p}{n}}$. Let us consider $\tilde{\alpha}$ obtained from (\[eq:double\]).

\(i) If $\pi_{j}^{*}=0$ for all $j\in K$, then there exist positive universal constants $c_{1},c_{2},c_{3},c_{4},c^{*},c_{0}^{*}$ such that $$\mathbb{P}\left(\left|\tilde{\alpha}-\alpha^{*}\right|\leq\overline{OVB}\vert\hat{I}_{1}=\hat{I}_{2}=\emptyset\right)\geq1-c_{1}k\exp\left(-c_{2}\log p\right)-c_{3}\exp\left(-c_{4}nr^{2}\right) \label{eq:OVB_upper}$$ where $\mathbb{P}\left(\hat{I}_{1}=\hat{I}_{2}=\emptyset\right)\geq1-c^{*}\exp\left(-c_{0}^{*}\log p\right)$ and $$\overline{OVB}\asymp\max\left\{ \left|\alpha^{*}\right|\left(\frac{k\log p}{n}\wedge1\right),\,\frac{\sigma_{v}\sigma_{\eta}\left(r\vee\frac{k\log p}{n}\right)}{\left(\frac{k\log p}{n}\vee1\right)\sigma_{v}^{2}}\right\}$$ for any $r\in(0,\,1]$.

\(ii) If $0<\left|\pi_{j}^{*}\right|\leq\frac{\lambda_{1}n}{2s}$ but $\left|\pi_{j}^{*}\right|\asymp\left(\sigma_{\eta}+\left|\alpha^{*}\right|\sigma_{v}\right)\sqrt{\frac{n}{s}}\sqrt{\frac{\log p}{n}}$ for all $j\in K$, then there exist positive universal constants $c_{1},c_{2},c_{3},c_{4},c^{*},c_{0}^{*}$ such that (\[eq:OVB\_upper\]) holds with $\mathbb{P}\left(\hat{I}_{1}=\hat{I}_{2}=\emptyset\right)\geq1-c^{*}\exp\left(-c_{0}^{*}\log p\right)$ and $$\overline{OVB}\asymp\max\left\{ \left(\left|\alpha^{*}\right|\vee\frac{\sigma_{\eta}}{\sigma_{v}}\right)\left(\frac{k\log p}{n}\wedge1\right),\,\frac{\sigma_{v}\sigma_{\eta}\left(r\vee\frac{k\log p}{n}\right)}{\left(\frac{k\log p}{n}\vee1\right)\sigma_{v}^{2}}\right\}$$ for any $r\in(0,\,1]$.

Suppose $\sigma_{v}\asymp1$ and $\frac{c^{'}k}{p^{c^{''}}}$ is small for some positive universal constants $c^{'}$ and $c^{''}$. As $\frac{k\log p}{n}\rightarrow\infty$ (where both Lasso steps are inconsistent in the sense that $\sqrt{\frac{1}{n}\sum_{i=1}^{n}\left(X_{i}\hat{\pi}-X_{i}\pi^{*}\right)^{2}}\rightarrow\infty$ and $\sqrt{\frac{1}{n}\sum_{i=1}^{n}\left(X_{i}\hat{\gamma}-X_{i}\gamma^{*}\right)^{2}}\rightarrow\infty$ with high probability), Proposition \[prop:bias\_post\_double\_selection-upper\] implies that $\overline{OVB}\asymp\frac{\sigma_{\eta}}{\sigma_{v}}$ and Proposition \[prop:bias\_post\_double\_selection-upper-1\] implies that $\overline{OVB}\asymp\left(\left|\alpha^{*}\right|\vee\frac{\sigma_{\eta}}{\sigma_{v}}\right)$, and $\mathbb{P}\left(\left|\tilde{\alpha}-\alpha^{*}\right|\leq\overline{OVB}\vert\hat{I}_{1}=\hat{I}_{2}=\emptyset\right)$ is large.

Implications for inference and comparison to high-dimensional OLS-based methods
===============================================================================

As shown in Section \[sec:inference\], the OVBs of post double Lasso have important consequences for making inference. In terms of simulation evidence, Figure \[fig:dml\_cov\] displays the coverage rates of 90% confidence intervals based on the DGP in Section \[sec:simulation\_evidence\] and shows that the OVB of post double Lasso can cause substantial under-coverage even when $n=1000$.

\[fig:dml\_cov\]

These results and those in Section \[sec:inference\] prompt the question of how to make inference in a reliable manner when under-selection and OVBs is a concern, such as, when limited variability is present. In many economic applications, $p$ is comparable to but still smaller than $n$. In such moderately high dimensional settings, OLS-based inference procedures provide a natural alternative to Lasso-based methods.

Under classical conditions, OLS is the best linear unbiased estimator. Moreover, under normality and homoscedasticity, OLS admits exact finite sample inference for any fixed $(n,\,p)$ as long as $\frac{p+1}{n}\leq1$ (recalling that the number of regression coefficients is $p+1$ in ).[^24] Furthermore, unlike the Lasso-based inference methods, OLS does not rely on any sparsity assumptions. This is important because sparsity assumptions may not be satisfied in applications and, as we show in this paper, the OVBs of Lasso-based inference procedures can be substantial even when $k$ is very small, $n$ is large and larger than $p$.

While OLS is unbiased, constructing standard errors is challenging when $p$ is large. For instance, @cattaneo2018inference show that the usual versions of Eicker-White robust standard errors are inconsistent under asymptotics where the number of controls grows as fast as the sample size. This result motivates a recent literature to develop high-dimensional OLS-based inference procedures that are valid in settings with many controls [e.g., @cattaneo2018inference; @jochmans2018heteroscedasticity; @kline2018leaveout].

Figures \[fig:dml\_bias\_std\_ols\]–\[fig:dml\_cov\_ols\] compare the finite sample performance of post double Lasso and OLS with the heteroscedasticity robust HCK standard errors proposed by @cattaneo2018inference. We do not report results for $n=200$ since OLS cannot be applied in this case. OLS is unbiased (as expected) and exhibits close-to-exact empirical coverage rates irrespective of the degree of variability in the controls. The additional simulation evidence in Appendix \[app:additional\_simulations\] confirms the excellent performance of OLS with HCK standard errors. Figure \[fig:dml\_ci\_ols\] displays the average length of 90% confidence intervals and shows that OLS yields somewhat wider confidence intervals than post double Lasso.

\[fig:dml\_bias\_std\_ols\]

\[fig:dml\_cov\_ols\]

\[fig:dml\_ci\_ols\]

In sum, our simulation results suggest that modern OLS-based inference methods that accommodate many controls constitute a viable alternative to Lasso-based inference methods. These methods are unbiased and demonstrate an excellent size accuracy, irrespective of the degree of variability in the relevant controls. However, there is a trade-off because OLS yields somewhat wider confidence intervals than post double Lasso.

Recommendations for empirical practice {#sec:recommendation}
======================================

Here we summarize the practical implications of our results and provide guidance for empirical researchers.

First, the simulation evidence in Section \[sec:evidence\] and Appendix \[app:additional\_simulations\] and the theoretical results in Section \[sec:bias\_post\_double\_lasso\] (cf. Remark \[rem:when\_post\_double\_work\]) suggest the following heuristic: If the estimates of $\alpha^{*}$ are robust to increasing the theoretically recommended regularization parameters in the two Lasso steps, post double Lasso could be a reliable and efficient method. Therefore, we recommend to always check whether empirical results are robust to increasing the regularization parameters. Based on our simulations, a simple rule of thumb is to increase by $50\%$ the regularization parameters proposed in @belloni2014inference. Robustness checks are standard in other contexts (e.g., bandwidth choices in regression discontinuity designs) and our results highlight the importance of such checks in the context of Lasso-based inference methods.

Second, we show that even if the relevant controls are associated with large coefficients, the Lasso may not select such controls due to limited variability. Therefore, following @belloni2014inference, we recommend to always augment the union of selected controls with an “amelioration” set of controls motivated by economic theory and prior knowledge to mitigate the OVBs.

Third, our simulations show that in moderately high-dimensional settings where $p$ is comparable to but smaller than $n$, recently developed OLS-based inference methods that are robust to the inclusion of many controls exhibit better size properties and not much wider confidence intervals than Lasso-based inference methods. This suggests that high-dimensional OLS-based procedures constitute a viable alternative to Lasso-based inference methods.

Conclusion
==========

Given the rapidly increasing popularity of Lasso and Lasso-based inference methods in empirical economic research, it is crucial to better understand the merits and limitations of these new tools, and how they compare to other alternatives such as the high-dimensional OLS-based procedures.

This paper presents empirical and theoretical evidence on the finite sample behavior of post double Lasso and the debiased Lasso (in the appendix). Specifically, we analyze the finite sample OVBs arising from the Lasso not selecting all the relevant control variables. Our results have important practical implications, and we provide guidance for empirical researchers.

We focus on the implications of under-selection for post double Lasso and the debiased Lasso in linear regression models. However, our results on the under-selection of the Lasso also have important implications for other inference methods that rely on Lasso as a first-step estimator. Towards this end, an interesting avenue for future research would be to investigate the impact of under-selection on the performance of the non-linear Lasso-based approaches proposed by @belloni2014inference, @farrell2015robust, @belloni2017program, and @chernozhukov2018double. In moderately high-dimensional settings where $p$ is smaller than but comparable to $n$, it would also be interesting to compare the non-linear treatment effects estimators in @belloni2014inference to the robust finite sample methods proposed by @rothe2017robust.

Finally, this paper motivates further examinations of the practical usefulness of Lasso-based inference procedures and other modern high-dimensional methods. For example, in the context of the study of elite college effects by @dale2002estimating, @angrist2019machine find that post double Lasso can be useful for regression-based sensitivity analyses when there is an abundance of potential control variables. They further present interesting simulation evidence on the finite sample behavior of Lasso-based IV methods [e.g., @belloni2012sparse].

[Appendix to “Omitted variable bias of Lasso-based inference methods: A finite sample analysis” (for online publication)]{}

Notation {#notation .unnumbered}
========

Here we collect additional notation that is not provided in the main text. The $\ell_{q}-$norm of a vector $v\in\mathbb{R}^{m}$ is denoted by $\left|v\right|_{q}$, $1\leq q\leq\infty$ where $\left|v\right|_{q}:=\left(\sum_{i=1}^{m}\left|v_{i}\right|^{q}\right)^{1/q}$ when $1\leq q<\infty$ and $\left|v\right|_{q}:=\max_{i=1,\dots,m}\left|v_{i}\right|$ when $q=\infty$. For a matrix $A\in\mathbb{R}^{n\times m}$, the $\ell_{2}-$operator norm of $A$ is defined as $\left\Vert A\right\Vert _{2}:=\sup_{v\in S^{m-1}}\left|Av\right|_{2}$, where $S^{m-1}=\left\{ v\in\mathbb{R}^{m}:\,\left|v\right|_{2}=1\right\} $. For a square matrix $A\in\mathbb{R}^{m\times m}$, let $\lambda_{\min}(A)$ and $\lambda_{\max}(A)$ denote its minimum eigenvalue and maximum eigenvalue, respectively.

Proofs for the main results
===========================

Lemma \[prop:fixed\_design\] {#appendix a1}
----------------------------

### Preliminary {#preliminary .unnumbered}

We will exploit the following Gaussian tail bound: $$\mathbb{P}\left(\mathcal{Z}\geq t\right)\leq\frac{1}{2}\exp\left(\frac{-t^{2}}{2\sigma^{2}}\right)$$ for all $t\geq0$, where $\mathcal{Z}\sim\mathcal{N}\left(0,\,\sigma^{2}\right)$. Note that the constant “$\frac{1}{2}$” cannot be improved uniformly.

Given $\lambda\geq\frac{2\sigma}{\phi}\sqrt{\frac{s}{n}}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}}$ where $\tau>0$ and the tail bound $$\mathbb{P}\left(\left|\frac{X^{T}\varepsilon}{n}\right|_{\infty}\geq t\right)\leq\exp\left(\frac{-nt^{2}}{2\sigma^{2}s/n}+\log p\right)\leq\frac{1}{p^{\tau}}$$ for $t=\frac{\sigma}{\phi}\sqrt{\frac{s}{n}}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}}$, we have $$\lambda\geq2\left|\frac{X^{T}\varepsilon}{n}\right|_{\infty}\label{eq:lambda}$$ with probability at least $1-\frac{1}{p^{\tau}}$. Let the event $$\mathcal{E}=\left\{ \left|\frac{X^{T}\varepsilon}{n}\right|_{\infty}\leq\frac{\sigma}{\phi}\sqrt{\frac{s}{n}}\sqrt{\frac{2\left(1+\tau\right)\log p}{n}}\right\} .\label{eq:event}$$ Note that $\mathbb{P}\left(\mathcal{E}\right)\geq1-\frac{1}{p^{\tau}}$.

Lemma \[prop:fixed\_design\] relies on the following intermediate results.

*(i) On the event $\mathcal{E}$, (\[eq:las\]) has a unique optimal solution $\hat{\theta}$ such that $\hat{\theta}_{j}=0$ for $j\notin K$.*

*(ii) If $\mathbb{P}\left(\left\{ \hat{\theta}_{j}\neq0,\,j\in K\right\} \cap\mathcal{E}\right)>0$, conditioning on $\left\{ \hat{\theta}_{j}\neq0,\,j\in K\right\} \cap\mathcal{E}$, we must have $$\left|\hat{\theta}_{j}-\theta_{j}^{*}\right|\geq\frac{\lambda n}{2s}.\label{eq:37-1}$$*

Claim (i) above follows from the argument in @wainwright_2019. To show claim (ii), we develop our own proof.

The proof for claim (i) above is based on a construction called Primal-Dual Witness (PDW) method developed by @wainwright2009sharp. The procedure is described as follows.

1.  Set $\hat{\theta}_{K^{c}}=0_{p-k}$.

2.  Obtain $(\hat{\theta}_{K},\,\hat{\delta}_{K})$ by solving $$\hat{\theta}_{K}\in\arg\min_{\theta_{K}\in\mathbb{R}^{k}}\left\{ \underset{:=g(\theta_{K})}{\underbrace{\frac{1}{2n}\left|Y-X_{K}\theta_{K}\right|_{2}^{2}}}+\lambda\left|\theta_{K}\right|_{1}\right\} ,\label{eq:sub}$$ and choosing $\hat{\delta}_{K}\in\partial\left|\theta_{K}\right|_{1}$ such that $\nabla g(\theta_{K})\vert_{\theta_{K}=\hat{\theta}_{K}}+\lambda\hat{\delta}_{K}=0$.[^25]

3.  Obtain $\hat{\delta}_{K^{c}}$ by solving $$\frac{1}{n}X^{T}(X\hat{\theta}-Y)+\lambda\hat{\delta}=0,\label{eq:42}$$ and check whether or not $\left|\hat{\delta}_{K^{c}}\right|_{\infty}<1$ (the *strict dual feasibility* condition) holds.

Lemma 7.23 from Chapter 7 of @wainwright_2019 shows that, if the PDW construction succeeds, then $\hat{\theta}=(\hat{\theta}_{K},\,0_{p-k})$ is the unique optimal solution of program (\[eq:las\]). To show that the PDW construction succeeds on the event $\mathcal{E}$, it suffices to show that $\left|\hat{\delta}_{K^{c}}\right|_{\infty}<1$. The details can be found in Chapter 7.5 of @wainwright_2019. In particular, under the choice of $\lambda$ stated in Lemma \[prop:fixed\_design\], we obtain that $\left|\hat{\delta}_{K^{c}}\right|_{\infty}<1$ and hence the PDW construction succeeds conditioning on $\mathcal{E}$ where $\mathbb{P}\left(\mathcal{E}\right)\geq1-\frac{1}{p^{\tau}}$.

In summary, conditioning on $\mathcal{E}$, under the choice of $\lambda$ stated in Lemma \[prop:fixed\_design\], program (\[eq:las\]) has a unique optimal solution $\hat{\theta}$ such that $\hat{\theta}_{j}=0$ for $j\notin K$.

We now show (\[eq:37-1\]). By construction, $\hat{\theta}=(\hat{\theta}_{K},\,0_{p-k})$, $\hat{\delta}_{K}$, and $\hat{\delta}_{K^{c}}$ satisfy (\[eq:42\]) and therefore we obtain $$\begin{aligned}
\frac{1}{n}X_{K}^{T}X_{K}\left(\hat{\theta}_{K}-\theta_{K}^{*}\right)-\frac{1}{n}X_{K}^{T}\varepsilon+\lambda\hat{\delta}_{K} & = & 0_{k},\label{eq:7-1}\\
\frac{1}{n}X_{K^{c}}^{T}X_{K}\left(\hat{\theta}_{K}-\theta_{K}^{*}\right)-\frac{1}{n}X_{K^{c}}^{T}\varepsilon+\lambda\hat{\delta}_{K^{c}} & = & 0_{p-k}.\label{eq:8}\end{aligned}$$ Solving the equations above yields $$\hat{\theta}_{K}-\theta_{K}^{*}=\left(\frac{X_{K}^{T}X_{K}}{n}\right)^{-1}\frac{X_{K}^{T}\varepsilon}{n}-\lambda\left(\frac{X_{K}^{T}X_{K}}{n}\right)^{-1}\hat{\delta}_{K}.\label{eq:49}$$

In what follows, we will condition on $\left\{ \hat{\theta}_{j}\neq0,\,j\in K\right\} \cap\mathcal{E}$ and make use of (\[eq:lambda\])-(\[eq:event\]). Let $\Delta=\frac{X_{K}^{T}\varepsilon}{n}-\lambda\hat{\delta}_{K}$. Note that $$\left|\hat{\theta}_{K}-\theta_{K}^{*}\right|\geq\left|\left(\frac{X_{K}^{T}X_{K}}{n}\right)^{-1}\right|\left|\left|\lambda\hat{\delta}_{K}\right|-\left|\frac{X_{K}^{T}\varepsilon}{n}\right|\right|,\label{eq:7}$$ where the inequality uses the fact that $\left(\frac{X_{K}^{T}X_{K}}{n}\right)^{-1}$ is diagonal. In Step 2 of the PDW procedure, $\hat{\delta}_{K}$ is chosen such that $\left|\hat{\delta}_{j}\right|=1$ for any $j\in K$ with $\hat{\theta}_{j}\neq0$; we therefore obtain $$\left|\hat{\theta}_{j}-\theta_{j}^{*}\right|\geq\frac{n}{s}\left|\left|\lambda\right|-\left|\frac{X_{j}^{T}\varepsilon}{n}\right|\right|\geq\frac{\lambda n}{2s}$$ where the second inequality follows from (\[eq:lambda\]).

### Main proof  {#main-proof .unnumbered}

In what follows, we let $$\begin{aligned}
    E_{1} & = & \left\{ \textrm{sgn}\left(\hat{\theta}_{j}\right)=-\textrm{sgn}\left(\theta_{j}^{*}\right),\,\textrm{for some }j\in K\right\} ,\\
    E_{2} & = & \left\{ \textrm{sgn}\left(\hat{\theta}_{j}\right)=\textrm{sgn}\left(\theta_{j}^{*}\right),\,\textrm{for some }j\in K\textrm{ such that (\ref{eq:min}) holds}\right\} ,\\
    E_{3} & = & \left\{ \textrm{sgn}\left(\hat{\theta}_{j}\right)=\textrm{sgn}\left(\theta_{j}^{*}\right),\,\textrm{for some }j\in K\right\} .\end{aligned}$$ To show (\[eq:nec\]) in (iv), recall we have established that conditioning on $\mathcal{E}$, (\[eq:las\]) has a unique optimal solution $\hat{\theta}$ such that $\hat{\theta}_{j}=0$ for $j\notin K$. Therefore, conditioning on $\mathcal{E}$, the KKT condition for (\[eq:las\]) implies $$\frac{s}{n}\left(\theta_{j}^{*}-\hat{\theta}_{j}\right)=\lambda\textrm{sgn}\left(\hat{\theta}_{j}\right)-\frac{X_{j}^{T}\varepsilon}{n}\label{eq:kkt}$$ for $j\in K$ such that $\hat{\theta}_{j}\neq0$.

We first show that $\mathbb{P}\left(E_{1}\cap\mathcal{E}\right)=0$. Suppose $\mathbb{P}\left(E_{1}\cap\mathcal{E}\right)>0$. We may then condition on the event $E_{1}\cap\mathcal{E}$. Case (i): $\theta_{j}^{*}>0$ and $\hat{\theta}_{j}<0$. Then, the LHS of (\[eq:kkt\]), $\frac{s}{n}\left(\theta_{j}^{*}-\hat{\theta}_{j}\right)>0$; consequently, the RHS, $\lambda\textrm{sgn}\left(\hat{\theta}_{j}\right)-\frac{X_{j}^{T}\varepsilon}{n}=-\lambda-\frac{X_{j}^{T}\varepsilon}{n}>0$. However, given the choice of $\lambda$, conditioning on $\mathcal{E}$, $\lambda\geq2\left|\frac{X^{T}\varepsilon}{n}\right|_{\infty}$ and consequently, $-\lambda-\frac{X_{j}^{T}\varepsilon}{n}\leq-\frac{\lambda}{2}<0$. This leads to a contradiction. Case (ii): $\theta_{j}^{*}<0$ and $\hat{\theta}_{j}>0$. Then, the LHS of (\[eq:kkt\]), $\frac{s}{n}\left(\theta_{j}^{*}-\hat{\theta}_{j}\right)<0$; consequently, the RHS, $\lambda\textrm{sgn}\left(\hat{\theta}_{j}\right)-\frac{X_{j}^{T}\varepsilon}{n}=\lambda-\frac{X_{j}^{T}\varepsilon}{n}<0$. However, given the choice of $\lambda$, conditioning on $\mathcal{E}$, $\lambda\geq2\left|\frac{X^{T}\varepsilon}{n}\right|_{\infty}$ and consequently, $\lambda-\frac{X_{j}^{T}\varepsilon}{n}\geq\frac{\lambda}{2}>0$. This leads to a contradiction.

It remains to show that $\mathbb{P}\left(E_{2}\cap\mathcal{E}\right)=0$. We first establish a useful fact under the assumption that $\mathbb{P}\left(E_{3}\cap\mathcal{E}\right)>0$. Let us condition on the event $E_{3}\cap\mathcal{E}$. If $\theta_{j}^{*}>0$, we have $\frac{s}{n}\left(\theta_{j}^{*}-\hat{\theta}_{j}\right)=\lambda-\frac{X_{j}^{T}\varepsilon}{n}\geq\frac{\lambda}{2}>0$ (i.e., $\theta_{j}^{*}\geq\hat{\theta}_{j}$); similarly, if $\theta_{j}^{*}<0$, then we have $\frac{s}{n}\left(\theta_{j}^{*}-\hat{\theta}_{j}\right)=-\lambda-\frac{X_{j}^{T}\varepsilon}{n}\leq-\frac{\lambda}{2}<0$ (i.e., $\theta_{j}^{*}\leq\hat{\theta}_{j}$). Putting the pieces together implies that, for $j\in K$ such that $\textrm{sgn}\left(\hat{\theta}_{j}\right)=\textrm{sgn}\left(\theta_{j}^{*}\right)$, $$\left|\theta_{j}^{*}-\hat{\theta}_{j}\right|=\left|\theta_{j}^{*}\right|-\left|\hat{\theta}_{j}\right|.\label{eq:equal}$$

We now show that $\mathbb{P}\left(E_{2}\cap\mathcal{E}\right)=0$. Suppose $\mathbb{P}\left(E_{2}\cap\mathcal{E}\right)>0$. We may then condition on the event that $E_{2}\cap\mathcal{E}$. Because of (\[eq:min\]) and (\[eq:equal\]), we have $\left|\theta_{j}^{*}-\hat{\theta}_{j}\right|<\frac{\lambda n}{2s}$. On the other hand, (\[eq:37-1\]) implies that $\left|\theta_{j}^{*}-\hat{\theta}_{j}\right|\geq\frac{\lambda n}{2s}$. We have arrived at a contradiction. Consequently, we must have $\mathbb{P}\left(E_{2}\cap\mathcal{E}\right)=0$.

In summary, we have shown that $\mathbb{P}\left(E_{1}\cap\mathcal{E}\right)=0$ and $\mathbb{P}\left(E_{2}\cap\mathcal{E}\right)=0$. Claim (i) in “Preliminary” implies that $\mathbb{P}\left(E_{4}\vert\mathcal{E}\right)=0$ where $E_{4}$ denotes the event that $\hat{\theta}_{j}\neq0$ for some $j\notin K$. Therefore, on $\mathcal{E}$, none of the events $E_{1}$, $E_{2}$ and $E_{4}$ can happen. This fact implies that, if (\[eq:min\]) is satisfied for all $l\in K$, we must have $$\mathbb{P}\left(\hat{\theta}=0_{p}\right)\geq1-\mathbb{P}\left(\mathcal{E}^{c}\right)\geq1-\frac{1}{p^{\tau}}.$$

Proposition \[prop:bias\_post\_double\_formula\] {#sec:appendix_post_double_lasso}
------------------------------------------------

We first show the case where $ab>0$. Let the events $$\begin{aligned}
\mathcal{E}_{t_{1}} & = & \left\{ \left|\frac{X_{K}^{T}v}{n}\right|_{\infty}\leq t_{1},\,t_{1}>0\right\} ,\label{eq:50}\\
\mathcal{E}_{t_{2}}^{'} & = & \left\{ \frac{1}{n}\sum_{i=1}^{n}v_{i}^{2}\leq\sigma_{v}^{2}+t_{2},\:t_{2}\in(0,\,\sigma_{v}^{2}]\right\} .\nonumber \end{aligned}$$ By tail bounds for Gaussian and Chi-Square variables, we have $$\begin{aligned}
\mathbb{P}\left(\mathcal{E}_{t_{1}}\right) & \geq & 1-k\exp\left(\frac{-nt_{1}^{2}}{2\frac{s}{n}\sigma_{v}^{2}}\right),\label{eq:32}\\
\mathbb{P}\left(\mathcal{E}_{t_{2}}^{'}\right) & \geq & 1-\exp\left(\frac{-nt_{2}^{2}}{8\sigma_{v}^{4}}\right).\nonumber \end{aligned}$$

In the following proof, we exploit the bound $$\begin{aligned}
\mathbb{P}\left(\mathcal{E}_{t_{2}}^{'}\vert\hat{I}_{2}=\emptyset,\,\mathcal{E}_{t_{1}}\right) & \geq & \mathbb{P}\left(\mathcal{E}_{t_{2}}^{'}\cap\mathcal{E}_{t_{1}}\cap\left\{ \hat{I}_{2}=\emptyset\right\} \right)\nonumber \\
& \geq & \mathbb{P}\left(\mathcal{E}_{t_{1}}\right)+\mathbb{P}\left(\mathcal{E}_{t_{2}}^{'}\right)+\mathbb{P}\left(\hat{I}_{2}=\emptyset\right)-2\nonumber \\
& \geq & 1-\frac{1}{p^{\tau}}-k\exp\left(\frac{-nt_{1}^{2}}{2\frac{s}{n}\sigma_{v}^{2}}\right)-\exp\left(\frac{-nt_{2}^{2}}{8\sigma_{v}^{4}}\right)\label{eq:58}\end{aligned}$$ where the third inequality follows from Lemma \[prop:fixed\_design\], which implies $\hat{I}_{2}=\emptyset$ with probability at least $1-\frac{1}{p^{\tau}}$. Note that $\mathbb{P}\left(\mathcal{E}_{t_{1}}\cap\left\{ \hat{I}_{2}=\emptyset\right\} \right)\geq\mathbb{P}\left(\mathcal{E}_{t_{1}}\right)+\mathbb{P}\left(\hat{I}_{2}=\emptyset\right)-1\geq1-\frac{1}{p^{\tau}}-k\exp\left(\frac{-nt_{1}^{2}}{2\frac{s}{n}\sigma_{v}^{2}}\right)$, which is a “high probability” guarantee for sufficiently large $p$ and $t_{1}$. Thus, working with $\mathbb{P}\left(\mathcal{E}_{t_{2}}^{'}\vert\hat{I}_{2}=\emptyset,\,\mathcal{E}_{t_{1}}\right)$ is sensible under an appropriate choice of $t_{1}$ (as we will see below).

We first bound $\frac{\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*}$. Note that $$\begin{aligned}
    \frac{\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*} & = & \left(\frac{D^{T}D}{n}\right)^{-1}\left[\frac{1}{n}\left(X_{K}\gamma_{K}^{*}+v\right)^{T}X_{K}\beta_{K}^{*}\right]\\
    & = & \left(\frac{D^{T}D}{n}\right)^{-1}\left[\frac{1}{n}\gamma_{K}^{*T}X_{K}^{T}X_{K}\beta_{K}^{*}+\frac{1}{n}v^{T}X_{K}\beta_{K}^{*}\right]\\
    & = & \frac{\frac{s}{n}\gamma_{K}^{*T}\beta_{K}^{*}+\frac{1}{n}v^{T}X_{K}\beta_{K}^{*}}{\frac{1}{n}\left(X_{K}\gamma_{K}^{*}+v\right)^{T}\left(X_{K}\gamma_{K}^{*}+v\right)},\end{aligned}$$ and that $\frac{s}{n}\gamma_{K}^{*T}\beta_{K}^{*}=2\left(1+\tau\right)ab\phi^{-2}\sigma_{\eta}\sigma_{v}\frac{k\log p}{n}$. Moreover, applying (\[eq:58\]) with $t_{1}=\left|b\right|\frac{\lambda_{2}}{4}$ and $t_{2}=r\sigma_{v}^{2}$ (where $r\in(0,\,1]$) yields

$$\begin{aligned}
\frac{s}{n}\gamma_{K}^{*T}\beta_{K}^{*}+\frac{1}{n}v^{T}X_{K}\beta_{K}^{*} & \geq & \frac{s}{n}\gamma_{K}^{*T}\beta_{K}^{*}-\left|\frac{1}{n}v^{T}X_{K}\right|_{\infty}\left|\beta_{K}^{*}\right|_{1}\nonumber \\
& \geq & \left(1+\tau\right)ab\phi^{-2}\sigma_{\eta}\sigma_{v}\frac{k\log p}{n}\label{eq:pos}\end{aligned}$$

as well as $$\begin{aligned}
\frac{1}{n}\left(X_{K}\gamma_{K}^{*}+v\right)^{T}\left(X_{K}\gamma_{K}^{*}+v\right) & \leq & \frac{s}{n}\gamma_{K}^{*T}\gamma_{K}^{*}+\left|\frac{2}{n}v^{T}X_{K}\right|_{\infty}\left|\gamma_{K}^{*}\right|_{1}+\frac{1}{n}v^{T}v\nonumber \\
& \leq & 4\left(1+\tau\right)\phi^{-2}b^{2}\sigma_{v}^{2}\frac{k\log p}{n}+\sigma_{v}^{2}+r\sigma_{v}^{2}\label{eq:den}\end{aligned}$$ with probability at least $$1-k\exp\left(\frac{-b^{2}\left(1+\tau\right)\log p}{4\phi^{2}}\right)-\frac{1}{p^{\tau}}-\exp\left(\frac{-nr^{2}}{8}\right):=T_{2}\left(r\right).$$

Conditioning on $\mathcal{E}_{t_{1}}\cap\left\{ \hat{I}_{2}=\emptyset\right\} $ with $t_{1}=t^{*}=\left|b\right|\frac{\lambda_{2}}{4}$, putting the pieces together yields $$\frac{D^{T}X_{K}}{D^{T}D}\beta_{K}^{*}\geq\frac{\left(1+\tau\right)ab\phi^{-2}\sigma_{\eta}\frac{k\log p}{n}}{4\left(1+\tau\right)\phi^{-2}b^{2}\sigma_{v}\frac{k\log p}{n}+\sigma_{v}+r\sigma_{v}}:=T_{1}\left(r\right),\label{eq:47}$$ with probability at least $T_{2}\left(r\right)$. That is, $$\mathbb{P}\left(\frac{D^{T}X_{K}}{D^{T}D}\beta_{K}^{*}\geq T_{1}\left(r\right)\vert\hat{I}_{2}=\emptyset,\,\mathcal{E}_{t^{*}}\right)\geq T_{2}\left(r\right).$$

When $\alpha^{*}=0$ in (\[eq:20\]), the reduced form coefficients $\pi^{*}$ in (\[eq:reduce\]) coincide with $\beta^{*}$ and $u$ coincides with $\eta$. Given the conditions on $X,$ $\eta$, $v$, $\beta_{K}^{*}$ and $\gamma_{K}^{*}$, we can then apply (\[eq:nec\]) in Lemma \[prop:fixed\_design\] and the fact $\mathbb{P}\left(\hat{I}_{1}=\hat{I}_{2}=\emptyset\right)\geq\mathbb{P}\left(\hat{I}_{1}=\emptyset\right)+\mathbb{P}\left(\hat{I}_{2}=\emptyset\right)-1$ to show that $E=\left\{ \hat{I}_{1}=\hat{I}_{2}=\emptyset\right\} $ occurs with probability at least $1-\frac{2}{p^{\tau}}$. Note that with the choice $t_{1}=t^{*}=\left|b\right|\frac{\lambda_{2}}{4}$, $\mathbb{P}\left(E\cap\mathcal{E}_{t^{*}}\right)\geq\mathbb{P}\left(E\right)+\mathbb{P}\left(\mathcal{E}_{t^{*}}\right)-1\geq1-k\exp\left(\frac{-b^{2}\left(1+\tau\right)\log p}{4\phi^{2}}\right)-\frac{2}{p^{\tau}}$, which is a “high probability” guarantee given sufficiently large $p$.[^26] Therefore, it is sensible to work with $\mathbb{E}\left(\tilde{\alpha}-\alpha^{*}\vert\mathcal{M}\right)$ where $$\mathcal{M}=E\cap\mathcal{E}_{t^{*}}.\label{eq:M_event}$$

Given $E$, (\[eq:double\]) becomes $$\tilde{\alpha}\in\textrm{arg}\min_{\alpha\in\mathbb{R}}\frac{1}{2n}\left|Y-D\alpha\right|_{2}^{2},\qquad\textrm{while }\tilde{\beta}=0_{p}.\label{eq:double-1}$$ As a result, we obtain $\mathbb{E}\left(\tilde{\alpha}-\alpha^{*}\vert\mathcal{M}\right)=\mathbb{E}\left(\frac{\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*}\vert\mathcal{M}\right)+\mathbb{E}\left(\frac{\frac{1}{n}D^{T}\eta}{\frac{1}{n}D^{T}D}\vert\mathcal{M}\right)$ and $$\begin{aligned}
\mathbb{E}\left(\frac{\frac{1}{n}D^{T}\eta}{\frac{1}{n}D^{T}D}\vert\mathcal{M}\right) & = & \frac{1}{\mathbb{P}\left(\mathcal{M}\right)}\mathbb{E}\left[\frac{\frac{1}{n}D^{T}\eta}{\frac{1}{n}D^{T}D}1_{\mathcal{M}}\left(D,\eta\right)\right]\nonumber \\
& = & \frac{1}{\mathbb{P}\left(\mathcal{M}\right)}\mathbb{E}_{D}\left\{ \mathbb{E}_{\eta}\left[\frac{\frac{1}{n}D^{T}\eta}{\frac{1}{n}D^{T}D}1_{\mathcal{M}}\left(D,\eta\right)\vert D\right]\right\} \nonumber \\
& = & \frac{1}{\mathbb{P}\left(\mathcal{M}\right)}\mathbb{E}_{D}\left\{ \frac{\frac{1}{n}\sum_{i=1}^{n}D_{i}\mathbb{E}_{\eta}\left[\eta_{i}1_{\mathcal{M}}\left(D,\eta\right)\vert D\right]}{\frac{1}{n}D^{T}D}\right\} \nonumber \\
& = & 0\label{eq:60}\end{aligned}$$ where $1_{\mathcal{M}}\left(D,\eta\right)=1\left\{ \left(v,\eta\right):\,\hat{I}_{1}=\hat{I}_{2}=\emptyset,\,\left|\frac{X_{K}^{T}v}{n}\right|_{\infty}\leq t^{*}\right\} $ (recall $X$ is a fixed design); the last line follows from $\frac{1}{n}\sum_{i=1}^{n}D_{i}=0$, the distributional identicalness of $\left(\eta_{i}\right)_{i=1}^{n}$ and that $\mathbb{E}_{\eta}\left[\eta_{i}1_{\mathcal{M}}\left(D,\eta\right)\vert D\right]$ is a constant over $i$s.

It remains to bound $\mathbb{E}\left(\frac{\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*}\vert\mathcal{M}\right)=\mathbb{E}\left(\frac{\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*}\vert\hat{I}_{2}=\emptyset,\,\mathcal{E}_{t^{*}}\right)$. Note that conditioning on $\mathcal{E}_{t^{*}}$, $\frac{\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*}$ is positive by (\[eq:pos\]). Applying a Markov inequality yields $$\mathbb{E}\left(\frac{\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*}\vert\hat{I}_{2}=\emptyset,\,\mathcal{E}_{t^{*}}\right)\geq T_{1}\left(r\right)\mathbb{P}\left(\frac{D^{T}X_{K}}{D^{T}D}\beta_{K}^{*}\geq T_{1}\left(r\right)\vert\hat{I}_{2}=\emptyset,\,\mathcal{E}_{t^{*}}\right)\geq T_{1}\left(r\right)T_{2}\left(r\right).$$ Combining the result above with (\[eq:60\]) and maximizing over $r\in(0,\,1]$ gives the claim.

We now show the case where $ab<0$. The argument is almost similar. In particular, we use $$\begin{aligned}
    \frac{s}{n}\gamma_{K}^{*T}\beta_{K}^{*}+\frac{1}{n}v^{T}X_{K}\beta_{K}^{*} & \leq & \frac{s}{n}\gamma_{K}^{*T}\beta_{K}^{*}+\left|\frac{1}{n}v^{T}X_{K}\right|_{\infty}\left|\beta_{K}^{*}\right|_{1}\\
    & \leq & -\left(1+\tau\right)\left|ab\right|\phi^{-2}\sigma_{\eta}\sigma_{v}\frac{k\log p}{n}<0\end{aligned}$$ and replace (\[eq:pos\]) with $$-\frac{s}{n}\gamma_{K}^{*T}\beta_{K}^{*}-\frac{1}{n}v^{T}X_{K}\beta_{K}^{*}\geq\left(1+\tau\right)\left|ab\right|\phi^{-2}\sigma_{\eta}\sigma_{v}\frac{k\log p}{n}.$$ Note that $$\mathbb{E}\left(\alpha^{*}-\tilde{\alpha}\vert\mathcal{M}\right)=\mathbb{E}\left(\frac{-\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*}\vert\mathcal{M}\right)-\mathbb{E}\left(\frac{\frac{1}{n}D^{T}\eta}{\frac{1}{n}D^{T}D}\vert\mathcal{M}\right).$$ So the rest of the proof follows from the argument for the case where $ab>0$.

Proposition \[prop:bias\_post\_double\_selection\] {#appendix:bias_post_double_lower_scaling}
--------------------------------------------------

Part (i) of Proposition \[prop:bias\_post\_double\_selection\] follows immediately from the proof for Proposition \[prop:bias\_post\_double\_formula\]. It remains to establish part (ii) where $\alpha^{*}\neq0$, $\alpha^{*}\gamma_{j}^{*}\in(0,\,-\beta_{j}^{*}]$, $\beta_{j}^{*}<0$ for all $j\in K$ (or, $\alpha^{*}\gamma_{j}^{*}\in[-\beta_{j}^{*},\,0)$, $\beta_{j}^{*}>0$ for all $j\in K$). Because of these conditions, we have $$\left|\pi_{j}^{*}\right|=\left|\beta_{j}^{*}+\alpha^{*}\gamma_{j}^{*}\right|<\left|\beta_{j}^{*}\right|\quad\forall j\in K.$$ Note that $\left|\alpha^{*}\right|\leq\max_{j\in K}\frac{\left|\beta_{j}^{\ast}\right|}{\left|\gamma_{j}^{\ast}\right|}\asymp\frac{\sigma_{\eta}}{\sigma_{v}}$ and $$\begin{aligned}
    \left|\frac{X^{T}u}{n}\right|_{\infty} & = & \left|\frac{X^{T}\left(\eta+\alpha^{*}v\right)}{n}\right|_{\infty}\\
    & \leq & \left|\frac{X^{T}\eta}{n}\right|_{\infty}+\left|\frac{\alpha^{*}X^{T}v}{n}\right|_{\infty}\\
    & \precsim & \sigma_{\eta}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}+\frac{\sigma_{\eta}}{\sigma_{v}}\sigma_{v}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}\\
    & \precsim & \phi^{-1}\sigma_{\eta}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}\end{aligned}$$ with probability at least $1-c_{1}^{'}\exp\left(-c_{2}^{'}\log p\right)$. The fact above justifies the choice of $\lambda_{1}$ stated in \[prop:bias\_post\_double\_selection\]. We can then apply (\[eq:nec\]) in Lemma \[prop:fixed\_design\] to show that $\hat{I}_{1}=\emptyset$ with probability at least $1-c_{5}\exp\left(-c_{6}\log p\right)$. Furthermore, under the conditions on $X$ and $\gamma_{K}^{*}$, (\[eq:nec\]) in Lemma \[prop:fixed\_design\] implies that $\hat{I}_{2}=\emptyset$ with probability at least $1-c_{0}\exp\left(-c_{0}^{'}\log p\right)$. Therefore, we have $$\mathbb{P}\left(\hat{I}_{1}=\hat{I}_{2}=\emptyset\right)\geq\mathbb{P}\left(\hat{I}_{1}=\emptyset\right)+\mathbb{P}\left(\hat{I}_{2}=\emptyset\right)-1\geq1-c_{1}^{''}\exp\left(-c_{2}^{''}\log p\right).$$

Given $u=\eta+\alpha^{*}v$, when $\alpha^{*}\neq0$, the event $\left\{ \hat{I}_{1}=\emptyset\right\} $ is not independent of $D$, so $\mathbb{E}\left(\frac{\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*}\vert E,\,\mathcal{E}_{t^{*}}\right)\neq\mathbb{E}\left(\frac{\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*}\vert\hat{I}_{2}=\emptyset,\,\mathcal{E}_{t^{*}}\right)$ (recalling $E=\left\{ \hat{I}_{1}=\hat{I}_{2}=\emptyset\right\} $). Instead of (\[eq:58\]), we apply

$$\begin{aligned}
    \mathbb{P}\left(\mathcal{E}_{t_{2}}^{'}\vert E,\,\mathcal{E}_{t_{1}}\right) & \geq & \mathbb{P}\left(\mathcal{E}_{t_{2}}^{'}\cap\mathcal{E}_{t_{1}}\cap E\right)\\
    & \geq & \mathbb{P}\left(\mathcal{E}_{t_{1}}\right)+\mathbb{P}\left(\mathcal{E}_{t_{2}}^{'}\right)+\mathbb{P}\left(E\right)-2\\
    & \geq & 1-c_{1}^{''}\exp\left(-c_{2}^{''}\log p\right)-k\exp\left(\frac{-nt_{1}^{2}}{2\frac{s}{n}\sigma_{v}^{2}}\right)\\
    &  & -\exp\left(\frac{-nt_{2}^{2}}{8\sigma_{v}^{4}}\right),\qquad\textrm{for any }t_{2}\in(0,\,\sigma_{v}^{2}].\end{aligned}$$

The rest of the proof follows from the argument for Proposition \[prop:bias\_post\_double\_formula\] and the bounds above.

Proposition \[prop:bias\_post\_double\_selection-1\]
----------------------------------------------------

Note that we have $$\begin{aligned}
    \left|\frac{X^{T}u}{n}\right|_{\infty} & = & \left|\frac{X^{T}\left(\eta+\alpha^{*}v\right)}{n}\right|_{\infty}\\
    & \leq & \left|\frac{X^{T}\eta}{n}\right|_{\infty}+\left|\frac{\alpha^{*}X^{T}v}{n}\right|_{\infty}\\
    & \precsim & \sigma_{\eta}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}+\left|\alpha^{*}\right|\sigma_{v}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}},\end{aligned}$$ which justifies the choice of $\lambda_{1}$ stated in Proposition \[prop:bias\_post\_double\_selection-1\].

For part (i), recall that $\pi_{K}^{*}=0_{k}$. By (i) of the intermediate results in “Preliminary” of Section \[appendix a1\], $\mathbb{P}\left(\hat{\pi}=0_{p}\right)=\mathbb{P}\left(\hat{I}_{1}=\emptyset\right)\geq1-c\exp\left(-c^{'}\log p\right)$. Substituting $\beta_{K}^{*}=-\alpha^{*}\gamma_{K}^{*}$ in $\frac{\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*}$ and following the rest of proof for Proposition \[prop:bias\_post\_double\_selection\] yields the claim.

For part (ii), for all $j\in K$, note that $\beta_{j}^{*}>-\alpha^{*}\gamma_{j}^{*}>0$ in case (1) and $\beta_{j}^{*}<-\alpha^{*}\gamma_{j}^{*}<0$ in case (2). Under the conditions on $X$ and $\pi_{K}^{*}$, (\[eq:nec\]) in Lemma \[prop:fixed\_design\] implies that $\mathbb{P}\left(\hat{\pi}=0_{p}\right)=\mathbb{P}\left(\hat{I}_{1}=\emptyset\right)\geq1-c\exp\left(-c^{'}\log p\right)$. Substituting $\beta_{K}^{*}>-\alpha^{*}\gamma_{K}^{*}>0$ for case (1) and $\beta_{j}^{*}<-\alpha^{*}\gamma_{j}^{*}<0$ for case (2) in the derivation of the bounds for $\frac{1}{n}D^{T}X_{K}\beta_{K}^{*}$ and following the rest of proof for Proposition \[prop:bias\_post\_double\_selection\] yields the claim.

Proposition \[prop:bias\_post\_double\_selection-upper\] \[appendix:Proposition-OVB-upper-proof\]
-------------------------------------------------------------------------------------------------

Given $\left\{ \hat{I}_{1}=\hat{I}_{2}=\emptyset\right\} $, note that $\left|\tilde{\alpha}-\alpha^{*}\right|\leq\left|\frac{\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*}\right|+\left|\frac{\frac{1}{n}D^{T}\eta}{\frac{1}{n}D^{T}D}\right|$. We make use of the following bound on Chi-Square variables: $$\mathbb{P}\left[\frac{1}{n}\sum_{i=1}^{n}v_{i}^{2}-\mathbb{E}\left(\frac{1}{n}\sum_{i=1}^{n}v_{i}^{2}\right)\leq-\sigma_{v}^{2}r^{'}\right]\leq\exp\left(\frac{-nr^{'2}}{16}\right)\label{eq:41}$$ for all $r^{'}\geq0$. On the event $\left\{ \hat{I}_{1}=\hat{I}_{2}=\emptyset\right\} $, choosing $t_{1}=t^{*}=\frac{\left|\gamma^{*}\right|s}{4n}$ in (\[eq:50\]) and $r^{'}=\frac{1}{2}$ in (\[eq:41\]) yields $$\begin{aligned}
    \left|\frac{\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*}\right| & \leq & \frac{\frac{s}{n}\left|\gamma_{K}^{*T}\beta_{K}^{*}\right|+\left|\beta_{K}^{*}\right|_{1}t^{*}}{\frac{s}{n}\gamma_{K}^{*T}\gamma_{K}^{*}-2\left|\gamma_{K}^{*}\right|_{1}t^{*}+\frac{1}{2}\sigma_{v}^{2}}\\
    & \leq & \frac{c_{3}\sigma_{\eta}\sigma_{v}\frac{k\log p}{n}}{c_{1}\frac{k\log p}{n}\sigma_{v}^{2}+c_{2}\sigma_{v}^{2}}\\
    & \leq & c_{4}\frac{\sigma_{\eta}}{\sigma_{v}}\left(\frac{k\log p}{n}\wedge1\right)\end{aligned}$$ with probability at least $1-c_{5}k\exp\left(-c_{6}\log p\right)-\exp\left(\frac{-n}{64}\right)$.

We can also show that $$\begin{aligned}
    &  & \mathbb{P}\left(\left|\frac{1}{n}D^{T}\eta\right|\leq t\vert\hat{I}_{1}=\hat{I}_{2}=\emptyset\right)\\
    & \geq & \mathbb{P}\left(\left\{ \left(\eta,v\right):\,\left|\frac{1}{n}D^{T}\eta\right|\leq t\right\} \cap\left\{ \hat{I}_{1}=\hat{I}_{2}=\emptyset\right\} \right)\\
    & \geq & \mathbb{P}\left(\left|\frac{1}{n}D^{T}\eta\right|\leq t\right)+\mathbb{P}\left(\hat{I}_{1}=\hat{I}_{2}=\emptyset\right)-1\\
    & \geq & 1-c_{5}\exp\left(-c_{6}\log p\right)-\mathbb{P}\left(\left|\frac{1}{n}D^{T}\eta\right|>t\right).\end{aligned}$$ Note that $\left|\frac{1}{n}D^{T}\eta\right|\leq k\left|\gamma^{*}\right|\left|\frac{1}{n}X^{T}\eta\right|_{\infty}+\left|\frac{1}{n}v^{T}\eta\right|$ where $$\begin{aligned}
    \mathbb{P}\left(\left|\frac{1}{n}v^{T}\eta\right|\precsim\sigma_{v}\sigma_{\eta}r\right) & \geq & 1-2\exp\left(-c_{7}nr^{2}\right)\quad\textrm{for any }r\in(0,\,1],\\
    \mathbb{P}\left(\left|\frac{1}{n}X^{T}\eta\right|_{\infty}\precsim\sigma_{\eta}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}\right) & \geq & 1-2\exp\left(-c_{8}\log p\right).\end{aligned}$$ The inequalities above yield $$\begin{aligned}
& \mathbb{P}\left\{ \left|\frac{1}{n}D^{T}\eta\right|\precsim\left[\left(\sigma_{v}\sigma_{\eta}r\right)\vee\underset{\asymp\sigma_{v}\sigma_{\eta}\frac{k\log p}{n}}{\underbrace{\left(k\left|\gamma^{*}\right|\sigma_{\eta}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}\right)}}\right]\vert\hat{I}_{1}=\hat{I}_{2}=\emptyset\right\} \\
\geq & 1-c_{5}\exp\left(-c_{6}\log p\right)-2\exp\left(-c_{7}nr^{2}\right)-2\exp\left(-c_{8}\log p\right).\end{aligned}$$ We have already shown that, conditioning on $\left\{ \hat{I}_{1}=\hat{I}_{2}=\emptyset\right\} $, $\frac{1}{n}D^{T}D\succsim\left(\frac{k\log p}{n}\vee1\right)\sigma_{v}^{2}$ with probability at least $1-c_{5}k\exp\left(-c_{6}\log p\right)-\exp\left(\frac{-n}{64}\right)$. As a consequence, $$\begin{aligned}
    &  & \mathbb{P}\left\{ \left|\frac{\frac{1}{n}D^{T}\eta}{\frac{1}{n}D^{T}D}\right|\precsim\frac{\sigma_{v}\sigma_{\eta}\left(r\vee\frac{k\log p}{n}\right)}{\left(\frac{k\log p}{n}\vee1\right)\sigma_{v}^{2}}\vert\hat{I}_{1}=\hat{I}_{2}=\emptyset\right\} \\
    & \geq & \mathbb{P}\left\{ \left|\frac{1}{n}D^{T}\eta\right|\precsim\sigma_{v}\sigma_{\eta}\left(r\vee\frac{k\log p}{n}\right)\,\,and\,\,\frac{1}{n}D^{T}D\succsim\left(\frac{k\log p}{n}\vee1\right)\sigma_{v}^{2}\vert\hat{I}_{1}=\hat{I}_{2}=\emptyset\right\} \\
    & \geq & \mathbb{P}\left\{ \left|\frac{1}{n}D^{T}\eta\right|\precsim\sigma_{v}\sigma_{\eta}\left(r\vee\frac{k\log p}{n}\right)\vert\hat{I}_{1}=\hat{I}_{2}=\emptyset\right\} \\
    &  & +\mathbb{P}\left\{ \frac{1}{n}D^{T}D\succsim\left(\frac{k\log p}{n}\vee1\right)\sigma_{v}^{2}\vert\hat{I}_{1}=\hat{I}_{2}=\emptyset\right\} -1\\
    & \geq & 1-c_{9}k\exp\left(-c_{10}\log p\right)-c_{12}\exp\left(-c_{11}nr^{2}\right).\end{aligned}$$

Putting the pieces above together yields $$\mathbb{P}\left(\left|\tilde{\alpha}-\alpha^{*}\right|\leq\overline{OVB}\vert\hat{I}_{1}=\hat{I}_{2}=\emptyset\right)\geq1-c_{1}^{'}k\exp\left(-c_{2}^{'}\log p\right)-c_{4}^{'}\exp\left(-c_{3}^{'}nr^{2}\right)$$ where $\overline{OVB}\asymp\max\left\{ \frac{\sigma_{\eta}}{\sigma_{v}}\left(\frac{k\log p}{n}\wedge1\right),\,\frac{\sigma_{v}\sigma_{\eta}\left(r\vee\frac{k\log p}{n}\right)}{\left(\frac{k\log p}{n}\vee1\right)\sigma_{v}^{2}}\right\} $.

Proposition \[prop:bias\_post\_double\_selection-upper-1\]  {#appendix:Proposition-OVB-upper-proof-1}
-----------------------------------------------------------

For part (i), substituting $\beta_{K}^{*}=-\alpha^{*}\gamma_{K}^{*}$ in $\frac{\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*}$ and following the rest of proof for Proposition \[prop:bias\_post\_double\_selection-upper\] yields the claim. For part (ii), note that $\left|\beta_{j}^{*}\right|\leq\left|\pi_{j}^{*}\right|+\left|\alpha^{*}\gamma_{j}^{*}\right|\precsim\left(\sigma_{\eta}+\left|\alpha^{*}\right|\sigma_{v}\right)\sqrt{\frac{n}{s}}\sqrt{\frac{\log p}{n}}$ for all $j\in K$. Substituting $\left|\beta_{j}^{*}\right|\precsim\left(\sigma_{\eta}+\left|\alpha^{*}\right|\sigma_{v}\right)\sqrt{\frac{n}{s}}\sqrt{\frac{\log p}{n}}$ in the derivation of the upper bound for $\left|\frac{\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*}\right|$ and following the rest of proof for Proposition \[prop:bias\_post\_double\_selection-upper\] yields the claim.

Debiased Lasso 
===============

In this section, we present theoretical and simulation results on the OVB of the debiased Lasso proposed by @vandergeer2014asymptotically.

Theoretical results {#sec:bias_debiased_lasso}
-------------------

The idea of debiased Lasso is to start with an initial Lasso estimate $\hat{\theta}=\left(\hat{\alpha},\,\hat{\beta}\right)$ of $\theta^{*}=\left(\alpha^{*},\,\beta^{*}\right)$ in equation (\[eq:main-y\]), where $$\left(\hat{\alpha},\,\hat{\beta}\right)\in\textrm{arg}\min_{\alpha\in\mathbb{R},\beta\in\mathbb{R}^{p}}\frac{1}{2n}\left|Y-D\alpha-X\beta\right|_{2}^{2}+\lambda_{1}\left(\left|\alpha\right|+\left|\beta\right|_{1}\right).\label{eq:21-2}$$ Given the initial Lasso estimator $\hat{\alpha}$, the debiased Lasso adds a correction term to $\hat{\alpha}$ to reduce the bias introduced by regularization. In particular, the debiased Lasso takes the form $$\tilde{\alpha}=\hat{\alpha}+\frac{\hat{\Omega}_{1}}{n}\sum_{i=1}^{n}Z_{i}^{T}\left(Y_{i}-Z_{i}\hat{\theta}\right),\label{eq:5-3}$$ where $Z_{i}=\left(D_{i},\,X_{i}\right)$ and $\hat{\Omega}_{1}$ is the first row of $\hat{\Omega}$, which is an approximate inverse of $\frac{1}{n}Z^{T}Z$, $Z=\left\{ Z_{i}\right\} _{i=1}^{n}$. Several different strategies have been proposed for constructing the approximate inverse $\hat{\Omega}$; see, for example, @javanmard2014confidence, @vandergeer2014asymptotically, and @Zhang_Zhang. We will focus on the widely used method proposed by @vandergeer2014asymptotically, which sets $$\begin{aligned}
    \hat{\Omega}_{1} & := & \hat{\tau}_{1}^{-2}\left(\begin{array}{cccc}
        1 & -\hat{\gamma}_{1} & \cdots & -\hat{\gamma}_{p}\end{array}\right),\\
    \hat{\tau}_{1}^{2} & := & \frac{1}{n}\left|D-X\hat{\gamma}\right|_{2}^{2}+\lambda_{2}\left|\hat{\gamma}\right|_{1},\end{aligned}$$ where $\hat{\gamma}$ is defined in (\[eq:las-2\]).

\[prop:bias\_debiased\_lasso\] Let Assumption \[ass:inference\_normality\] and part (i) and (iii) of Assumption \[ass:inference\_sparsity\_incoherence\] hold. Suppose: with probability at least $1-\kappa$, $\left\Vert \left(Z_{-K}^{T}X_{K}\right)\left(X_{K}^{T}X_{K}\right)^{-1}\right\Vert _{\infty}\leq1-\frac{\phi}{2}$ for some $\phi\in(0,\,1]$ such that $\phi^{-1}\precsim1$, where $Z_{-K}$ denotes the columns in $Z=\left(D,\,X\right)$ excluding $X_{K}$; the regularization parameters in (\[eq:las-2\]) and (\[eq:21-2\]) are chosen in a similar fashion as in Lemma \[prop:fixed\_design\] such that $\lambda_{1}\asymp\phi^{-1}\left(\sqrt{\frac{s}{n}}\vee\sigma_{v}\right)\sigma_{\eta}\sqrt{\frac{\log p}{n}}$ and $\lambda_{2}\asymp\phi^{-1}\sigma_{v}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}$; for all $j\in K$, $\left|\beta_{j}^{*}\right|\leq\frac{\lambda_{1}n}{2s}$ and $\left|\gamma_{j}^{*}\right|\leq\frac{\lambda_{2}n}{2s}$, but $\left|\beta_{j}^{*}\right|\asymp\left[\sqrt{\frac{n}{s}}\vee\frac{n\sigma_{v}}{s}\right]\sigma_{\eta}\sqrt{\frac{\log p}{n}}$ and $\left|\gamma_{j}^{*}\right|\asymp\sigma_{v}\sqrt{\frac{n}{s}}\sqrt{\frac{\log p}{n}}$. Let us consider $\tilde{\alpha}$ obtained from (\[eq:5-3\]). If $\alpha^{*}=0$, then there exist positive universal constants $c^{\dagger},c_{1},c_{2},c_{3},c^{*},c_{0}^{*}$ such that $$\left|\mathbb{E}\left(\tilde{\alpha}-\alpha^{*}\vert\mathcal{M}^{'}\right)\right|\geq c^{\dagger}\frac{\sigma_{\eta}}{\sigma_{v}}\left(\frac{k\log p}{n}\wedge1\right)\left[1-2\kappa-c_{1}k\exp\left(-c_{2}\log p\right)-\exp\left(-c_{3}n\right)\right],$$ where $\mathcal{M}^{'}$ is an event with $\mathbb{P}\left(\mathcal{M}^{'}\right)\geq1-2\kappa-c^{*}k\exp\left(-c_{0}^{*}\log p\right)$.

\[prop:bias\_debiased\_lasso-upper\] Let Assumption \[ass:inference\_normality\] and part (i) and (iii) of Assumption \[ass:inference\_sparsity\_incoherence\] hold. Suppose: with probability at least $1-\kappa$, $\left\Vert \left(Z_{-K}^{T}X_{K}\right)\left(X_{K}^{T}X_{K}\right)^{-1}\right\Vert _{\infty}\leq1-\frac{\phi}{2}$ for some $\phi\in(0,\,1]$ such that $\phi^{-1}\precsim1$, where $Z_{-K}$ denotes the columns in $Z=\left(D,\,X\right)$ excluding $X_{K}$; the regularization parameters in (\[eq:las-2\]) and (\[eq:21-2\]) are chosen in a similar fashion as in Proposition \[prop:bias\_debiased\_lasso\] such that $\lambda_{1}\asymp\phi^{-1}\left(\sqrt{\frac{s}{n}}\vee\sigma_{v}\right)\sigma_{\eta}\sqrt{\frac{\log p}{n}}$ and $\lambda_{2}\asymp\phi^{-1}\sigma_{v}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}$; for all $j\in K$, $\gamma_{j}^{*}=\gamma^{*}$, $\left|\beta_{j}^{*}\right|\leq\frac{\lambda_{1}n}{2s}$ and $\left|\gamma^{*}\right|\leq\frac{\lambda_{2}n}{2s}$, but $\left|\beta_{j}^{*}\right|\asymp\left[\sqrt{\frac{n}{s}}\vee\frac{n\sigma_{v}}{s}\right]\sigma_{\eta}\sqrt{\frac{\log p}{n}}$ and $\left|\gamma^{*}\right|\asymp\sigma_{v}\sqrt{\frac{n}{s}}\sqrt{\frac{\log p}{n}}$. Let us consider $\tilde{\alpha}$ obtained from (\[eq:5-3\]). If $\alpha^{*}=0$, then there exist positive universal constants $c_{1},c_{2},c_{3},c_{4},c^{*},c_{0}^{*}$ such that $$\mathbb{P}\left(\left|\tilde{\alpha}-\alpha^{*}\right|\leq\overline{OVB}\vert \hat{\theta}=0_{p+1},\,\hat{\gamma}=0_{p} \right)\geq1-c_{1}k\exp\left(-c_{2}\log p\right)-c_{3}\exp\left(-c_{4}nr^{2}\right)$$ where $\mathbb{P}\left( \hat{\theta}=0_{p+1},\,\hat{\gamma}=0_{p} \right)\geq1-2\kappa-c^{*}k\exp\left(-c_{0}^{*}\log p\right)$ and $$\overline{OVB}\asymp\max\left\{ \frac{\sigma_{\eta}}{\sigma_{v}}\left(\frac{k\log p}{n}\wedge1\right),\,\frac{\sigma_{v}\sigma_{\eta}\left(r\vee\frac{k\log p}{n}\right)}{\left(\frac{k\log p}{n}\vee1\right)\sigma_{v}^{2}}\right\}$$ for any $r\in(0,\,1]$.

One can show that a population version of the mutual incoherence condition, $\left\Vert \left[\mathbb{E}\left(Z_{-K}^{T}\right)X_{K}\right]\left(X_{K}^{T}X_{K}\right)^{-1}\right\Vert _{\infty}=1-\phi$, implies $\left\Vert \left(Z_{-K}^{T}X_{K}\right)\left(X_{K}^{T}X_{K}\right)^{-1}\right\Vert _{\infty}\leq1-\frac{\phi}{2}$ with high probability (that is, $\kappa$ is small and vanishes polynomially in $p$). For example, we can apply (\[eq:77\]) in Lemma \[lem:A3\] with slight notational changes.

The event $\mathcal{M}^{'}$ in Proposition \[prop:bias\_debiased\_lasso\] is the intersection of $\left\{ \hat{\theta}=0_{p+1},\,\hat{\gamma}=0_{p}\right\}$ and an additional event, both of which occur with high probabilities. The additional event is needed in our analyses for technical reasons. See Appendix \[sec:appendix\_debiased\_lasso\] for details.

Proof for Propositions \[prop:bias\_debiased\_lasso\] and \[prop:bias\_debiased\_lasso-upper\] {#sec:appendix_debiased_lasso}
----------------------------------------------------------------------------------------------

Under the conditions in Proposition \[prop:bias\_debiased\_lasso\], (\[eq:nec\]) in Lemma \[prop:fixed\_design\] implies that $\hat{\gamma}=0_{p}$ with probability at least $1-c_{0}\exp\left(-c_{0}^{'}\log p\right)$. Conditioning on this event, $\hat{\Omega}_{1}=\left(\frac{1}{n}D^{T}D\right)^{-1}e_{1}$ where $e_{1}=\left(\begin{array}{cccc}
1 & 0 & \cdots & 0\end{array}\right)$. If $\alpha^{*}=0$, under the conditions in Proposition \[prop:bias\_debiased\_lasso\], we show that $\hat{\theta}=0_{p+1}$ with probability at least $1-2\kappa-c_{5}\exp\left(-c_{6}\log p\right)$. To achieve this goal, we slightly modify the argument for (\[eq:nec\]) in Lemma \[prop:fixed\_design\] by replacing (\[eq:event\]) with $\mathcal{E}=\mathcal{E}_{1}\cap\mathcal{E}_{2}$, where

$$\begin{aligned}
    \mathcal{E}_{1} & = & \left\{ \left|\frac{Z^{T}\eta}{n}\right|_{\infty}\precsim\phi^{-1}\left(\sqrt{\frac{s}{n}}\vee\sigma_{v}\right)\sigma_{\eta}\sqrt{\frac{\log p}{n}}\right\} ,\\
    \mathcal{E}_{2} & = & \left\{ \left\Vert \left(Z_{-K}^{T}X_{K}\right)\left(X_{K}^{T}X_{K}\right)^{-1}\right\Vert _{\infty}\leq1-\frac{\phi}{2}\right\} ,\end{aligned}$$

and $Z_{-K}$ denotes the columns in $Z$ excluding $X_{K}$. Note that by (\[eq:A37-1\]), $\mathbb{P}\left(\mathcal{E}_{1}\right)\geq1-c_{1}^{'}\exp\left(-c_{2}^{'}\log p\right)$ and therefore, $\mathbb{P}\left(\mathcal{E}\right)\geq1-\kappa-c_{1}^{'}\exp\left(-c_{2}^{'}\log p\right)$. We then follow the argument used in the proof for Lemma \[prop:fixed\_design\] to show $\mathbb{P}\left(E_{1}\cap\mathcal{E}\right)=0$ and $\mathbb{P}\left(E_{2}\cap\mathcal{E}\right)=0$, where $$\begin{aligned}
    E_{1} & = & \left\{ \textrm{sgn}\left(\hat{\beta}_{j}\right)=-\textrm{sgn}\left(\beta_{j}^{*}\right),\,\textrm{for some }j\in K\right\} ,\\
    E_{2} & = & \left\{ \textrm{sgn}\left(\hat{\beta}_{j}\right)=\textrm{sgn}\left(\beta_{j}^{*}\right),\,\textrm{for some }j\in K\right\} .\end{aligned}$$ Moreover, conditioning on $\mathcal{E}$, $\hat{\alpha}=0$ and $\hat{\beta}_{K^{c}}=0_{p-k}$. Putting these facts together yield the claim that $\hat{\theta}=0_{p+1}$ with probability at least $1-2\kappa-c_{5}\exp\left(-c_{6}\log p\right)$.

Letting $E=\left\{ \hat{\theta}=0_{p+1},\,\hat{\gamma}=0_{p}\right\} $ with $\mathbb{P}\left(E\right)\geq1-2\kappa-c_{1}\exp\left(-c_{2}\log p\right)$ and recalling the event $\mathcal{E}_{t^{*}}$ in the proof for Proposition \[prop:bias\_post\_double\_formula\], we can then show $$\begin{aligned}
    \mathbb{E}\left(\tilde{\alpha}-\alpha^{*}\vert\mathcal{M}^{'}\right) & = & \frac{1}{n}\mathbb{E}\left(\hat{\Omega}_{1}Z^{T}\eta\vert\mathcal{M}^{'}\right)+\mathbb{E}\left[\frac{D^{T}X_{K}}{D^{T}D}\left(\beta_{K}^{*}-\hat{\beta}_{K}\right)\vert\mathcal{M}^{'}\right]\\
    & = & \mathbb{E}\left(\frac{\frac{1}{n}D^{T}\eta}{\frac{1}{n}D^{T}D}\vert\mathcal{M}^{'}\right)+\mathbb{E}\left(\frac{D^{T}X_{K}}{D^{T}D}\beta_{K}^{*}\vert\mathcal{M}^{'}\right)\\
    & = & \mathbb{E}\left(\frac{D^{T}X_{K}}{D^{T}D}\beta_{K}^{*}\vert\mathcal{M}^{'}\right)\end{aligned}$$ where $\mathcal{M}^{'}=E\cap\mathcal{E}_{t^{*}}$ such that $\mathbb{P}\left(\mathcal{M}^{'}\right)\geq1-2\kappa-c_{3}^{*}k\exp\left(-c_{4}^{*}\log p\right)$ and the last line follows from the argument used to show (\[eq:60\]).

The rest of argument is similar to what is used in showing Proposition \[prop:bias\_post\_double\_formula\]. However, because (\[eq:21-2\]) involves $D$, $\mathbb{E}\left(\frac{\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*}\vert E,\,\mathcal{E}_{t^{*}}\right)\neq\mathbb{E}\left(\frac{\frac{1}{n}D^{T}X_{K}}{\frac{1}{n}D^{T}D}\beta_{K}^{*}\vert\hat{\gamma}=0_{p},\,\mathcal{E}_{t^{*}}\right)$. Instead of (\[eq:50\]) and (\[eq:58\]), we apply $$\begin{aligned}
    \mathbb{P}\left(\mathcal{E}_{t_{2}}^{'}\vert E,\,\mathcal{E}_{t_{1}}\right) & \geq & \mathbb{P}\left(\mathcal{E}_{t_{2}}^{'}\cap\mathcal{E}_{t_{1}}\cap E\right)\\
    & \geq & \mathbb{P}\left(\mathcal{E}_{t_{1}}\right)+\mathbb{P}\left(\mathcal{E}_{t_{2}}^{'}\right)+\mathbb{P}\left(E\right)-2\\
    & \geq & 1-2\kappa-c_{1}\exp\left(-c_{2}\log p\right)-k\exp\left(\frac{-nt_{1}^{2}}{2\frac{s}{n}\sigma_{v}^{2}}\right)\\
    &  & -\exp\left(\frac{-nt_{2}^{2}}{8\sigma_{v}^{4}}\right),\qquad\textrm{for any }t_{2}\in(0,\,\sigma_{v}^{2}].\end{aligned}$$ Consequently, we have the claim in Proposition \[prop:bias\_debiased\_lasso\].

Following the argument used to show Proposition \[prop:bias\_post\_double\_selection-upper\], we also have the claim in Proposition \[prop:bias\_debiased\_lasso-upper\].

Simulations evidence
--------------------

Here we evaluate the performance of the debiased Lasso proposed by @vandergeer2014asymptotically based on the simulation setting of Section \[sec:simulation\_evidence\]. We use cross-validation to choose the regularization parameters as it is the most commonly-used method in this literature. Figures \[fig:db\_bias\_std\] and \[fig:db\_cov\] present the results. Debiased Lasso exhibits substantial biases (relative to the standard deviation) and under-coverage for all values of $\sigma_x$ and its performance is very sensitive to the regularization choice. The performance tends to get worse as $\sigma_x$ increases because the bias decays slower than the standard deviation. A comparison to the results in Section \[sec:simulation\_evidence\] shows that post double Lasso performs better than debiased Lasso.[^27]

\[fig:db\_bias\_std\]

\[fig:db\_cov\]

Additional simulations for post double Lasso {#app:additional_simulations}
============================================

In the main text, we consider a setting with $k=5$, normally distributed control variables, normally distributed homoscedastic errors terms, and $\alpha^\ast=0$. Here we provide additional simulation evidence based on a more general model:[^28] $$\begin{aligned}
Y_{i} & = & D_{i}\alpha^{*}+X_{i}\beta^{*}+\sigma_{y}(D_{i},X_{i})\eta_{i},\label{eq:dgp_illustration_inference1_general}\\
D_{i} & = & X_{i}\gamma^{*}+\sigma_{d}(X_{i})v_{i},\label{eq:dgp_illustration_inference2_general}\end{aligned}$$ where $\eta_{i}$ and $v_{i}$ are independent of each other and $\left\{ X_{i},\eta_{i},v_{i}\right\} _{i=1}^{n}$ consists of i.i.d. entries. The object of interest is $\alpha^{*}$. We set $n=500$, $p=200$, and consider a sparse setting where $\beta^{*}=\gamma^\ast=(\underbrace{1,\dots,1}_{k},0,\dots,0)^{T}$. We consider six DGPs that differ with respect to $k$, the distributions of $X_i$, $\eta_{i}$, and $v_{i}$, the specifications of $\sigma_{y}(D_{i},X_{i})$ and $\sigma_{d}(X_{i})$, as well as $\alpha^\ast$. For DGP A1, we do not report the results for $\sigma_x< 0.2$ due to numerical issues with the computation of standard errors. The results are based on 1,000 simulation repetitions.

                                          $X_i$                                 $k$                                                  $\sigma_y(D_i,X_i)$                                                                                    $\sigma_d(X_i)$                                              $\eta_i$                        $v_i$               $\alpha^\ast$     
  -------- ------------------------------------------------------------------- ----- ------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- ----------------------------- ------------------------------ --------------- -- --
  DGP A1    Indep. $\text{Bern}\left(\frac{1-\sqrt{1-4\sigma_x^2}}{2}\right)$    5                                                            1                                                                                                    1                                                $ \mathcal{N}(0,1)$           $ \mathcal{N}(0,1)$              0           
  DGP A2                     $\mathcal{N}(0,\sigma_x^2I_p)$                      5                                                            1                                                                                                    1                                            $\frac{t(5)}{\sqrt{(5/3)}}$   $ \frac{t(5)}{\sqrt{(5/3)}}$         0           
  DGP A3                     $\mathcal{N}(0,\sigma_x^2I_p)$                      5    $\sqrt{\frac{(1+D_{i}\alpha^{*}+X_{i}\beta^{*})^{2}}{\frac{1}{n}\sum_{i}(1+D_{i}\alpha^{*}+X_{i}\beta^{*})^{2}}}$   $\sqrt{\frac{(1+X_{i}\gamma^{*})^{2}}{\frac{1}{n}\sum_{i}(1+X_{i}\gamma^{*})^{2}}}$       $ \mathcal{N}(0,1)$           $ \mathcal{N}(0,1)$              0           
  DGP A4                     $\mathcal{N}(0,\sigma_x^2I_p)$                     10                                                            1                                                                                                    1                                                $ \mathcal{N}(0,1)$           $ \mathcal{N}(0,1)$              0           
  DGP A5                     $\mathcal{N}(0,\sigma_x^2I_p)$                      5                                                            1                                                                                                    1                                                $ \mathcal{N}(0,1)$           $ \mathcal{N}(0,1)$              1           
  DGP A6                     $\mathcal{N}(0,\sigma_x^2I_p)$                      5                                                            1                                                                                                    1                                                $ \mathcal{N}(0,1)$           $ \mathcal{N}(0,1)$             -1           

Figures \[fig:dml\_bias\_std\_app\]–\[fig:dml\_ci\_app\] present the results. The two most important determinants of the performance of post double Lasso are $k$ and $\alpha^{*}$. To see why $\alpha^\ast$ is important, recall that the reduced form parameter and the error term in the first step of post double Lasso (i.e., program (\[eq:las-1\])) are $\pi^{*}=\alpha^{*}\gamma^{*}+\beta^{*}$ and $u_{i}=\eta_{i}+\alpha^{*}v_{i}$. This implies that the magnitude of $\pi^{*}$ as well as the variance of $u_i$ depend on $\alpha^{*}$. Consequently, the selection performance of Lasso in the first step is directly affected by $\alpha^{*}$. In the extreme case where $\alpha^{*}$ is such that $\pi^{*}=0_{p}$, Lasso does not select any controls with high probability if the regularization parameter is chosen according to the standard recommendations. The simulation results further show that there is no practical recommendation for choosing the regularization parameters. While $\lambda_{\text{min}}$ leads to the best performance when $\alpha^\ast=0$, this choice can yield poor performances when $\alpha^\ast> 0$. Finally, across all DGPs, OLS outperforms post double Lasso in terms of bias and coverage accuracy, but yields somewhat wider confidence intervals.

\[fig:dml\_bias\_std\_app\]

\[fig:dml\_cov\_app\]

\[fig:dml\_ci\_app\]

Theoretical results for random designs \[sec:Random-design\]
============================================================

Results
-------

In this section, we provide some results in Lemma \[prop:random\_design\] for the Lasso with a random design $X$. The necessary result on the Lasso’s inclusion established in Lemma \[prop:random\_design\] can be adopted in a similar fashion as in Propositions \[prop:bias\_post\_double\_selection\]-\[prop:bias\_post\_double\_selection-upper-1\] and \[prop:bias\_debiased\_lasso\]-\[prop:bias\_debiased\_lasso-upper\] to establish the OVBs.

We make the following assumption about (\[eq:1\]).

\[ass:random\_design\] Each row of $X$ is sampled independently; for all $i=1,\dots,n$ and $j=1,\dots,p$, $\sup_{r\geq1}r^{-\frac{1}{2}}\left(\mathbb{E}\left|X_{ij}\right|^{r}\right)^{\frac{1}{r}}\leq\alpha<\infty$; for any unit vector $a\in\mathbb{R}^{k}$ and $i=1,\dots,n$, $\sup_{r\geq1}r^{-\frac{1}{2}}\left(\mathbb{E}\left|a^{T}X_{i,K}^{T}\right|^{r}\right)^{\frac{1}{r}}\leq\tilde{\alpha}<\infty$, where $X_{i,K}$ is the $i$th row of $X_{K}$ and $K=\left\{ j:\,\theta_{j}^{*}\neq0\right\} $. Moreover, the error terms $\varepsilon_{1},\dots,\varepsilon_{n}$ are independent such that $\sup_{r\geq1}r^{-\frac{1}{2}}\left(\mathbb{E}\left|\varepsilon_{i}\right|^{r}\right)^{\frac{1}{r}}\leq\sigma<\infty$ and $\mathbb{E}\left(X_{i}\varepsilon_{i}\right)=0_{p}$ for all $i=1,\dots,n$.

Assumption \[ass:random\_design\] is known as the sub-Gaussian tail condition defined in @vershynin2012. Examples of sub-Gaussian variables include Gaussian mixtures and distributions with bounded support. The first and last part of Assumption \[ass:random\_design\] imply that $X_{ij},j=1,\dots,p,$ and $\varepsilon_{i}$ are sub-Gaussian variables and is used in deriving the lower bounds on the regularization parameters. The second part of Assumption \[ass:random\_design\] is only used to establish some eigenvalue condition on $\frac{X_{K}^{T}X_{K}}{n}$.

\[ass:sparsity\_incoherence\_random\_design\] The following conditions are satisfied: (i) $\theta^{*}$ is exactly sparse with at most $k$ non-zero coefficients and $K\neq\emptyset$; (ii) $$\left\Vert \left[\mathbb{E}\left(X_{K^{c}}^{T}X_{K}\right)\right]\left[\mathbb{E}\left(X_{K}^{T}X_{K}\right)\right]^{-1}\right\Vert _{\infty}=1-\phi\label{eq:14-1}$$ for some $\phi\in(0,\,1]$ such that $\phi^{-1}\precsim1$; (iii) $\mathbb{E}\left(X_{ij}\right)=0$ for all $j\in K$ and $\mathbb{E}\left(X_{j}^{T}X_{j}\right)\leq s$ for all $j=1,\dots,p$; (iv) $$\begin{aligned}
    \max\left\{ \frac{\phi}{12(1-\phi)k^{\frac{3}{2}}},\,\frac{\phi}{6k^{\frac{3}{2}}},\,\frac{\phi}{k}\right\} \sqrt{\frac{\log p}{n}} & \leq & \alpha^{2}\quad\textrm{if }\phi\in\left(0,\,1\right),\label{eq:7-2}\\
    \max\left\{ \frac{1}{6k^{\frac{3}{2}}},\,\frac{1}{k}\right\} \sqrt{\frac{\log p}{n}} & \leq & \alpha^{2}\quad\textrm{if }\phi=1,\label{eq:7-4}\\
    \max\left\{ 2\tilde{\alpha}^{2},\,12\alpha^{2},\,1\right\} \sqrt{\frac{\log p}{n}} & \leq & \lambda_{\min}\left(\mathbb{E}\left[\frac{1}{n}X_{K}^{T}X_{K}\right]\right).\label{eq:7-3}
    \end{aligned}$$

Part (iv) of Assumption \[ass:sparsity\_incoherence\_random\_design\] is imposed to ensure that $$\begin{aligned}
    \left\Vert \left(\frac{1}{n}X_{K}^{T}X_{K}\right)^{-1}-\left[\mathbb{E}\left(\frac{1}{n}X_{K}^{T}X_{K}\right)\right]^{-1}\right\Vert _{\infty} & \precsim & \frac{1}{\lambda_{\min}\left(\mathbb{E}\left[\frac{1}{n}X_{K}^{T}X_{K}\right]\right)},\\
    \left\Vert \frac{1}{n}X_{K^{c}}^{T}X_{K}\left(\frac{1}{n}X_{K}^{T}X_{K}\right)^{-1}\right\Vert _{\infty} & \leq & 1-\frac{\phi}{2},\end{aligned}$$ with high probability. To gain some intuition for , let us further assume $k\asymp1$, $X_{i}$ is normally distributed for all $i=1,\dots,n$, and $\mathbb{E}\left(X_{K}^{T}X_{K}\right)$ is a diagonal matrix with the diagonal entries $\mathbb{E}\left(X_{j}^{T}X_{j}\right)=s\neq0$. As a result, $\tilde{\alpha}=\alpha\asymp\sqrt{\frac{s}{n}}$ by the definition of a sub-Gaussian variable (e.g., @vershynin2012) and essentially require $\sqrt{\frac{\log p}{n}}\precsim\frac{s}{n}$.

Given $$\mathbb{P}\left(\left|\frac{X^{T}\varepsilon}{n}\right|_{\infty}\geq t\right)\leq2\exp\left(\frac{-nt^{2}}{c_{0}\sigma^{2}\alpha^{2}}+\log p\right).\label{eq:38}$$ and $\lambda\geq\frac{c\alpha\sigma\left(2-\frac{\phi}{2}\right)}{\phi}\sqrt{\frac{\log p}{n}}$ for some sufficiently large universal constant $c>0$, we have $$\lambda\geq2\left|\frac{X^{T}\varepsilon}{n}\right|_{\infty}\label{eq:lambda-1}$$ with probability at least $1-c^{'}\exp\left(-c^{''}\log p\right)$.

Define the following events $$\begin{aligned}
    \mathcal{E}_{1} & = & \left\{ \left|\frac{X^{T}\varepsilon}{n}\right|_{\infty}\precsim\frac{\alpha\sigma\left(2-\frac{\phi}{2}\right)}{\phi}\sqrt{\frac{\log p}{n}}\right\} ,\\
    \mathcal{E}_{2} & = & \left\{ \lambda_{\max}(\hat{\Sigma}_{KK})\leq\frac{3}{2}\lambda_{\max}(\Sigma_{KK})\right\} ,\\
    \mathcal{E}_{3} & = & \left\{ \left\Vert \left(\frac{1}{n}X_{K}^{T}X_{K}\right)^{-1}-\left[\mathbb{E}\left(\frac{1}{n}X_{K}^{T}X_{K}\right)\right]^{-1}\right\Vert _{\infty}\precsim\frac{1}{\lambda_{\min}\left(\mathbb{E}\left[\frac{1}{n}X_{K}^{T}X_{K}\right]\right)}\right\} ,\\
    \mathcal{E}_{4} & = & \left\{ \left\Vert \frac{1}{n}X_{K^{c}}^{T}X_{K}\left(\frac{1}{n}X_{K}^{T}X_{K}\right)^{-1}\right\Vert _{\infty}\leq1-\frac{\phi}{2}\right\} .\end{aligned}$$ By (\[eq:38\]), $\mathbb{P}\left(\mathcal{E}_{1}\right)\geq1-c^{'}\exp\left(-c^{''}\log p\right)$; by (\[eq:S7\]), $\mathbb{P}\left(\mathcal{E}_{2}\right)\geq1-c_{1}^{'}\exp\left(-c_{1}^{''}\log p\right)$; by (\[eq:S4\]), $\mathbb{P}\left(\mathcal{E}_{3}\right)\geq1-c_{2}^{'}\exp\left(-c_{2}^{''}\left(\frac{\log p}{k^{3}}\right)\right)$; by (\[eq:77\]), $\mathbb{P}\left(\mathcal{E}_{4}\right)\geq1-c_{3}^{''}\exp\left(-b\left(\frac{\log p}{k^{3}}\right)\right)$, where $b$ is some positive constant that only depends on $\phi$ and $\alpha$.

\[prop:random\_design\] Let Assumptions \[ass:random\_design\] and \[ass:sparsity\_incoherence\_random\_design\] hold. We solve the Lasso (\[eq:las\]) with $\lambda\geq\frac{c\alpha\sigma\left(2-\frac{\phi}{2}\right)}{\phi}\sqrt{\frac{\log p}{n}}$ for some sufficiently large universal constant $c>0$. Suppose $\mathbb{E}\left[X_{K}^{T}X_{K}\right]$ is a positive definite matrix.

\(i) Then, conditioning on $\mathcal{E}_{1}\cap\mathcal{E}_{4}$ (which holds with probability at least $1-c_{1}\exp\left(-b\frac{\log p}{k^{3}}\right)$), (\[eq:las\]) has a unique optimal solution $\hat{\theta}$ such that $\hat{\theta}_{j}=0$ for $j\notin K$.

\(ii) With probability at least $1-c_{1}\exp\left(-b\frac{\log p}{k^{3}}\right)$, $$\left|\hat{\theta}_{K}-\theta_{K}^{*}\right|_{2}\leq\frac{3\lambda\sqrt{k}}{\lambda_{\min}\left(\mathbb{E}\left[\frac{1}{n}X_{K}^{T}X_{K}\right]\right)}\label{eq:5gen-1}$$ where $\theta_{K}=\left\{ \theta_{j}\right\} _{j\in K}$ and $b$ is some positive constant that only depends on $\phi$ and $\alpha$; if $\mathbb{P}\left(\left\{ \textrm{supp}(\hat{\theta})=K\right\} \cap\mathcal{E}_{1}\cap\mathcal{E}_{2}\right)>0$, conditioning on $\left\{ \textrm{supp}(\hat{\theta})=K\right\} \cap\mathcal{E}_{1}\cap\mathcal{E}_{2}$, we must have $$\left|\hat{\theta}_{K}-\theta_{K}^{*}\right|_{2}\geq\frac{\lambda\sqrt{k}}{3\lambda_{\max}\left(\mathbb{E}\left[\frac{1}{n}X_{K}^{T}X_{K}\right]\right)}\geq\frac{\lambda\sqrt{k}}{3\sum_{j\in K}\left(\mathbb{E}\left[\frac{1}{n}X_{j}^{T}X_{j}\right]\right)}.\label{eq:4gen-1}$$

\(iii) If $\mathbb{E}\left(X_{K}^{T}X_{K}\right)$ is a diagonal matrix with the diagonal entries $\mathbb{E}\left(X_{j}^{T}X_{j}\right)=s\neq0$, then $$\left|\hat{\theta}_{j}-\theta_{j}^{*}\right|\leq\frac{7\lambda n}{4s}\qquad\forall j\in K\label{eq:5-1}$$ with probability at least $1-c_{1}\exp\left(-b\frac{\log p}{k^{3}}\right)$; if $\mathbb{P}\left(\left\{ \hat{\theta}_{j}\neq0,\,j\in K\right\} \cap\mathcal{E}_{1}\cap\mathcal{E}_{3}\cap\mathcal{E}_{4}\right)>0$, conditioning on $\left\{ \hat{\theta}_{j}\neq0,\,j\in K\right\} \cap\mathcal{E}_{1}\cap\mathcal{E}_{3}\cap\mathcal{E}_{4}$, we must have $$\left|\hat{\theta}_{j}-\theta_{j}^{*}\right|\geq\frac{\lambda n}{4s}\geq c_{0}\frac{\sigma}{\phi}\sqrt{\frac{n}{s}}\sqrt{\frac{\log p}{n}}.\label{eq:4-1}$$

\(iv) Suppose $K=\left\{ 1\right\} $ and $\mathbb{E}\left(X_{1}^{T}X_{1}\right)=s\neq0$. If $$\left|\theta_{1}^{*}\right|\leq\frac{\lambda n}{4s},\label{eq:min-1}$$ then we must have $$\mathbb{P}\left(\hat{\theta}=0_{p}\right)\geq1-c\exp\left(-b\log p\right).\label{eq:nec1}$$

The part $\frac{\lambda n}{4s}\geq c_{0}\frac{\sigma}{\phi}\sqrt{\frac{n}{s}}\sqrt{\frac{\log p}{n}}$ in bound (\[eq:4-1\]) follows from the fact that $\alpha\succsim\sqrt{\frac{s}{n}}=\sqrt{\mathbb{E}\left(\frac{1}{n}\sum X_{ij}^{2}\right)}$ where $j\in K$.

Main proof for Lemma \[prop:random\_design\]
--------------------------------------------

In what follows, we let $\Sigma_{KK}:=\mathbb{E}\left[\frac{1}{n}X_{K}^{T}X_{K}\right]$, $\hat{\Sigma}_{KK}:=\frac{1}{n}X_{K}^{T}X_{K}$, and $\lambda_{\min}\left(\Sigma\right)$ denote the minimum eigenvalue of the matrix $\Sigma$. The proof for Proposition \[prop:random\_design\](i) follows similar argument as before but requires a few extra steps. In applying Lemma 7.23 from Chapter 7.5 of @wainwright_2019 to establish the uniqueness of $\hat{\theta}$ upon the success of PDW construction, it suffices to show that $\lambda_{\min}(\hat{\Sigma}_{KK})\geq\frac{1}{2}\lambda_{\min}(\Sigma_{KK})$ and this fact is verified in (\[eq:S7-1\]) in the appendix. As a consequence, the subproblem (\[eq:sub\]) is strictly convex and has a unique minimizer. The details that show the PDW construction succeeds conditioning on $\mathcal{E}_{1}\cap\mathcal{E}_{4}$ (which holds with probability at least $1-c_{1}\exp\left(-b\frac{\log p}{k^{3}}\right)$) can be found in Lemma \[lem:A4\] (where $b$ is some positive constant that only depends on $\phi$ and $\alpha$).

To show (\[eq:5gen-1\]), note that our choice of $\lambda$ and $\left|\hat{\delta}_{K}\right|\leq1$ yield $$\left|\Delta\right|\leq\left|\lambda\hat{\delta}_{K}\right|+\left|\frac{X_{K}^{T}\varepsilon}{n}\right|\leq\frac{3\lambda}{2}1_{k},$$ which implies that $\left|\Delta\right|_{2}\leq\frac{3\lambda}{2}\sqrt{k}$. Moreover, we can show $$\begin{aligned}
\left|\hat{\theta}_{K}-\theta_{K}^{*}\right|_{2} & = & \frac{\left|\left(\frac{X_{K}^{T}X_{K}}{n}\right)^{-1}\Delta\right|_{2}}{\left|\Delta\right|_{2}}\left|\Delta\right|_{2}\label{eq:30-1}\\
& \leq & \frac{1}{\lambda_{\min}\left(\frac{1}{n}X_{K}^{T}X_{K}\right)}\frac{3\lambda}{2}\sqrt{k}.\nonumber \end{aligned}$$ Applying (\[eq:S7-1\]) and the bound $\left|\Delta\right|_{2}\leq\frac{3\lambda}{2}\sqrt{k}$ yields the claim.

In showing (\[eq:4gen-1\]) in (ii) and (\[eq:4-1\]) in (iii), we will condition on $\left\{ \textrm{supp}(\hat{\theta})=K\right\} \cap\mathcal{E}_{1}\cap\mathcal{E}_{2}$ and $\left\{ \hat{\theta}_{j}\neq0,\,j\in K\right\} \cap\mathcal{E}_{1}\cap\mathcal{E}_{3}\cap\mathcal{E}_{4}$, respectively.

To show (\[eq:4gen-1\]), note that in Step 2 of the PDW procedure, $\hat{\delta}_{K}$ is chosen such that $\left|\hat{\delta}_{j}\right|=1$ for any $j\in K$ whenever $\textrm{supp}(\hat{\theta})=K$. Given the choice of $\lambda$, we are ensured to have $$\left|\Delta\right|\geq\left|\left|\lambda\hat{\delta}_{K}\right|-\left|\frac{X_{K}^{T}\varepsilon}{n}\right|\right|\geq\frac{\lambda}{2}1_{k},$$ which implies that $\left|\Delta\right|_{2}\geq\frac{\lambda}{2}\sqrt{k}$. Moreover, we can show $$\left|\hat{\theta}_{K}-\theta_{K}^{*}\right|_{2}=\frac{\left|\left(\frac{X_{K}^{T}X_{K}}{n}\right)^{-1}\Delta\right|_{2}}{\left|\Delta\right|_{2}}\left|\Delta\right|_{2}\geq\frac{1}{\lambda_{\max}\left(\frac{1}{n}X_{K}^{T}X_{K}\right)}\frac{\lambda}{2}\sqrt{k}.\label{eq:30}$$ It remains to bound $\lambda_{\max}\left(\hat{\Sigma}_{KK}\right)$. We first write $$\begin{aligned}
    \lambda_{\max}(\Sigma_{KK}) & = & \max_{||h^{'}||_{2}=1}\mu^{'T}\Sigma_{KK}\mu^{'}\\
    & = & \max_{||h^{'}||_{2}=1}\left[\mu^{'T}\hat{\Sigma}_{KK}\mu^{'}+\mu^{'T}(\Sigma_{KK}-\hat{\Sigma}_{KK})\mu^{'}\right]\\
    & \geq & \mu^{T}\hat{\Sigma}_{KK}\mu+\mu^{T}(\Sigma_{KK}-\hat{\Sigma}_{KK})\mu\end{aligned}$$ where $\mu\in\mathbb{R}^{k}$ is a unit-norm maximal eigenvector of $\hat{\Sigma}_{KK}$. Applying Lemma \[lem:A1\](b) with $t=\tilde{\alpha}^{2}\sqrt{\frac{\log p}{n}}$ yields $$\mu^{T}\left(\Sigma_{KK}-\hat{\Sigma}_{KK}\right)\mu\geq-\tilde{\alpha}^{2}\sqrt{\frac{\log p}{n}}$$ with probability at least $1-c_{1}\exp\left(-c_{2}\log p\right)$, provided that $\sqrt{\frac{\log p}{n}}\leq1$; therefore, $\lambda_{\max}(\Sigma_{KK})\geq\lambda_{\max}(\hat{\Sigma}_{KK})-\tilde{\alpha}^{2}\sqrt{\frac{\log p}{n}}$. Because $\tilde{\alpha}^{2}\sqrt{\frac{\log p}{n}}\leq\frac{\lambda_{\max}(\Sigma_{KK})}{2}$ (implied by (\[eq:7-3\])), we have $$\lambda_{\max}(\hat{\Sigma}_{KK})\leq\frac{3}{2}\lambda_{\max}(\Sigma_{KK})\label{eq:S7}$$ with probability at least $1-c_{1}\exp\left(-c_{2}\log p\right)$.

As a consequence,

$$\left|\hat{\theta}_{K}-\theta_{K}^{*}\right|_{2}\geq\frac{1}{\lambda_{\max}\left(\frac{1}{n}X_{K}^{T}X_{K}\right)}\frac{\lambda}{2}\sqrt{k}\geq\frac{1}{\lambda_{\max}(\Sigma_{KK})}\frac{\lambda}{3}\sqrt{k}.$$ The second inequality in (\[eq:4gen-1\]) simply follows from the fact $\lambda_{\max}\left(\mathbb{E}\left[\frac{1}{n}X_{K}^{T}X_{K}\right]\right)\leq\sum_{j\in K}\left(\mathbb{E}\left[\frac{1}{n}X_{j}^{T}X_{j}\right]\right)$.

To show (\[eq:5-1\]), note that $$\begin{aligned}
\left|\hat{\theta}_{K}-\theta_{K}^{*}\right|_{\infty} & \leq & \left|\hat{\Sigma}_{KK}^{-1}\frac{X_{K}^{T}\varepsilon}{n}\right|_{\infty}+\lambda\left\Vert \hat{\Sigma}_{KK}^{-1}\right\Vert _{\infty}\nonumber \\
& \leq & \left\Vert \hat{\Sigma}_{KK}^{-1}\right\Vert _{\infty}\left|\frac{X_{K}^{T}\varepsilon}{n}\right|_{\infty}+\lambda\left\Vert \hat{\Sigma}_{KK}^{-1}\right\Vert _{\infty}\nonumber \\
& \leq & \frac{3\lambda}{2}\left\Vert \hat{\Sigma}_{KK}^{-1}\right\Vert _{\infty}.\label{eq:10-1}\end{aligned}$$ We then apply (\[eq:S4\]) of Lemma \[lem:A2\] in the appendix, and the fact $\left\Vert \hat{\Sigma}_{KK}^{-1}\right\Vert _{\infty}-\left\Vert \Sigma_{KK}^{-1}\right\Vert _{\infty}\leq\left\Vert \hat{\Sigma}_{KK}^{-1}-\Sigma_{KK}^{-1}\right\Vert _{\infty}$ (so that $\left\Vert \hat{\Sigma}_{KK}^{-1}\right\Vert _{\infty}\leq\frac{7n}{6s}$); putting everything yields the claim.

To show (\[eq:4-1\]), we again carry over the argument in the proof for Lemma \[prop:fixed\_design\]. Letting $M=\hat{\Sigma}_{KK}^{-1}-\Sigma_{KK}^{-1}$, we have $$\begin{aligned}
    \left|\hat{\theta}_{K}-\theta_{K}^{*}\right| & = & \left|\left(\Sigma_{KK}^{-1}+M\right)\left[\left(\frac{X_{K}^{T}\varepsilon}{n}\right)-\lambda\hat{\delta}_{K}\right]\right|\\
    & \geq & \left|\Sigma_{KK}^{-1}\left[\left(\frac{X_{K}^{T}\varepsilon}{n}\right)+\lambda\hat{\delta}_{K}\right]\right|-\left|M\left[\left(\frac{X_{K}^{T}\varepsilon}{n}\right)-\lambda\hat{\delta}_{K}\right]\right|\\
    & \geq & \left|\Sigma_{KK}^{-1}\right|\left|\left|\lambda\hat{\delta}_{K}\right|-\left|\frac{X_{K}^{T}\varepsilon}{n}\right|\right|-\left\Vert M\right\Vert _{\infty}\left|\left(\frac{X_{K}^{T}\varepsilon}{n}\right)-\lambda\hat{\delta}_{K}\right|_{\infty}1_{k},\end{aligned}$$ where the third line uses the fact that $\Sigma_{KK}^{-1}$ is diagonal.

Note that as before, the choice of $\lambda$ stated in Lemma \[prop:random\_design\] and the fact $\Sigma_{KK}^{-1}=\frac{n}{s}I_{k}$ yield $$\begin{aligned}
    \left|\hat{\theta}_{j}-\theta_{j}^{*}\right| & \geq & \frac{\lambda n}{2s}-\left\Vert M\right\Vert _{\infty}\left|\left(\frac{X_{K}^{T}\varepsilon}{n}\right)-\lambda\hat{\delta}_{K}\right|_{\infty}\\
    & \geq & \frac{\lambda n}{2s}-\frac{3}{2}\lambda\left\Vert M\right\Vert _{\infty}.\end{aligned}$$ By (\[eq:S4\]) of Lemma \[lem:A2\] in the appendix, with probability at least $1-c_{1}\exp\left(-b\frac{\log p}{k^{3}}\right)$, $\left\Vert M\right\Vert _{\infty}\leq\frac{1}{6}\lambda_{\min}^{-1}(\Sigma_{KK})=\frac{n}{6s}$.

As a result, we have (\[eq:4-1\]). The part $\frac{\lambda n}{4s}\geq c_{0}\frac{\sigma}{\phi}\sqrt{\frac{n}{s}}\sqrt{\frac{\log p}{n}}$ in bound (\[eq:4-1\]) follows from the fact that $\alpha\succsim\sqrt{\frac{s}{n}}=\sqrt{\mathbb{E}\left(\frac{1}{n}\sum X_{ij}^{2}\right)}$ where $j\in K$.

To establish (\[eq:nec1\]), we adopt argument similar to what is used in showing (\[eq:nec\]) by applying the KKT condition $$\left(\frac{1}{n}X_{1}^{T}X_{1}\right)\left(\theta_{1}^{*}-\hat{\theta}_{1}\right)=\lambda\textrm{sgn}\left(\hat{\theta}_{1}\right)-\frac{X_{1}^{T}\varepsilon}{n}$$ and defining $\mathcal{E}=\mathcal{E}_{1}\cap\mathcal{E}_{4}$.

Additional technical lemmas and proofs
--------------------------------------

In this section, we show that the PDW construction succeeds with high probability in Lemma \[lem:A4\], which is proved using results from Lemmas \[lem:A1\]\[lem:A3\]. The derivations for Lemmas \[lem:A2\] and \[lem:A3\] modify the argument in @wainwright2009sharp and @ravikumar2010 to make it suitable for our purposes. In what follows, we let $\Sigma_{K^{c}K}:=\mathbb{E}\left[\frac{1}{n}X_{K^{c}}^{T}X_{K}\right]$ and $\hat{\Sigma}_{K^{c}K}:=\frac{1}{n}X_{K^{c}}^{T}X_{K}$. Similarly, let $\Sigma_{KK}:=\mathbb{E}\left[\frac{1}{n}X_{K}^{T}X_{K}\right]$ and $\hat{\Sigma}_{KK}:=\frac{1}{n}X_{K}^{T}X_{K}$.

\[lem:A1\] (a) Let $\left(W_{i}\right)_{i=1}^{n}$ and $\left(W_{i}^{'}\right)_{i=1}^{n}$ consist of independent components, respectively. Suppose there exist parameters $\alpha$ and $\alpha^{'}$ such that $$\begin{aligned}
        \sup_{r\geq1}r^{-\frac{1}{2}}\left(\mathbb{E}\left|W_{i}\right|^{r}\right)^{\frac{1}{r}} & \leq & \alpha,\\
        \sup_{r\geq1}r^{-\frac{1}{2}}\left(\mathbb{E}\left|W_{i}^{'}\right|^{r}\right)^{\frac{1}{r}} & \leq & \alpha^{'},
    \end{aligned}$$ for all $i=1,\dots,n$. Then $$\mathbb{P}\left[\left|\frac{1}{n}\sum_{i=1}^{n}\left(W_{i}W_{i}^{'}\right)-\mathbb{E}\left[\frac{1}{n}\sum_{i=1}^{n}\left(W_{i}W_{i}^{'}\right)\right]\right|\geq t\right]\leq2\exp\left(-cn\left(\frac{t^{2}}{\alpha^{2}\alpha^{'2}}\wedge\frac{t}{\alpha\alpha^{'}}\right)\right).\label{eq:A37-1}$$ (b) For any unit vector $v\in\mathbb{R}^{d}$, suppose there exists a parameter $\tilde{\alpha}$ such that $$\sup_{r\geq1}r^{-\frac{1}{2}}\left(\mathbb{E}\left|a^{T}Z_{i}^{T}\right|^{r}\right)^{\frac{1}{r}}\leq\tilde{\alpha},$$ where $Z_{i}$ is the $i$th row of $Z\in\mathbb{R}^{n\times d}$, then we have $$\mathbb{P}(\left|\left|Zv\right|_{2}^{2}-\mathbb{E}\left(\left|Zv\right|_{2}^{2}\right)\right|\geq nt)\leq2\exp\left(-c^{'}n\left(\frac{t^{2}}{\tilde{\alpha}^{4}}\wedge\frac{t}{\tilde{\alpha}^{2}}\right)\right).$$

Lemma \[lem:A1\] is based on Lemma 5.14 and Corollary 5.17 in @vershynin2012.

\[lem:A2\] Suppose Assumption \[ass:random\_design\] holds. For any $t>0$ and some constant $c>0$, we have

$$\mathbb{P}\left\{ \left\Vert \hat{\Sigma}_{K^{c}K}-\Sigma_{K^{c}K}\right\Vert _{\infty}\geq t\right\} \leq2(p-k)k\exp\left(-cn\left(\frac{t^{2}}{k^{2}\alpha^{4}}\wedge\frac{t}{k\alpha^{2}}\right)\right),\label{eq:S2}$$

$$\mathbb{P}\left\{ \left\Vert \hat{\Sigma}_{KK}-\Sigma_{KK}\right\Vert _{\infty}\geq t\right\} \leq2k^{2}\exp\left(-cn\left(\frac{t^{2}}{k^{2}\alpha^{4}}\wedge\frac{t}{k\alpha^{2}}\right)\right).\label{eq:S3}$$

Furthermore, if $k\geq1$, $\frac{\log p}{n}\leq1$, $\tilde{\alpha}^{2}\sqrt{\frac{\log p}{n}}\leq\frac{\lambda_{\min}(\Sigma_{KK})}{2}$, and $\alpha^{2}\sqrt{\frac{\log p}{n}}\leq\frac{\lambda_{\min}(\Sigma_{KK})}{12}$, we have

$$\begin{aligned}
    \mathbb{P}\left\{ \left\Vert \hat{\Sigma}_{KK}^{-1}\right\Vert _{2}\leq\frac{2}{\lambda_{\min}(\Sigma_{KK})}\right\}  & \geq & 1-c_{1}^{'}\exp\left(-c_{2}^{'}\log p\right),\label{eq:S8}\\
    \mathbb{P}\left\{ \left\Vert \hat{\Sigma}_{KK}^{-1}-\Sigma_{KK}^{-1}\right\Vert _{\infty}\leq\frac{1}{6\lambda_{\min}(\Sigma_{KK})}\right\}  & \geq & 1-c_{1}\exp\left(-c_{2}\left(\frac{\log p}{k^{3}}\right)\right).\label{eq:S4}
    \end{aligned}$$

Let $u_{j^{'}j}$ denote the element $(j^{'},\,j)$ of the matrix difference $\hat{\Sigma}_{K^{c}K}-\Sigma_{K^{c}K}$. The definition of the $l_{\infty}$matrix norm implies that $$\begin{aligned}
        \mathbb{P}\left\{ \left\Vert \hat{\Sigma}_{K^{c}K}-\Sigma_{K^{c}K}\right\Vert _{\infty}\geq t\right\}  & = & \mathbb{P}\left\{ \max_{j^{'}\in K^{c}}\sum_{j\in K}|u_{j^{'}j}|\geq t\right\} \\
        & \leq & (p-k)\mathbb{P}\left\{ \sum_{j\in K}|u_{j^{'}j}|\geq t\right\} \\
        & \leq & (p-k)\mathbb{P}\left\{ \exists j\in K\,\vert\,|u_{j^{'}j}|\geq\frac{t}{k}\right\} \\
        & \leq & (p-k)k\mathbb{P}\left\{ |u_{j^{'}j}|\geq\frac{t}{k}\right\} \\
        & \leq & (p-k)k\cdot2\exp\left(-cn\left(\frac{t^{2}}{k^{2}\alpha^{4}}\wedge\frac{t}{k\alpha^{2}}\right)\right),
    \end{aligned}$$ where the last inequality follows Lemma \[lem:A1\](a). Bound (\[eq:S3\]) can be derived in a similar fashion except that the pre-factor $(p-k)$ is replaced by $k$.

To prove (\[eq:S4\]), note that

$$\begin{aligned}
    \left\Vert \hat{\Sigma}_{KK}^{-1}-\Sigma_{KK}^{-1}\right\Vert _{\infty} & = & \left\Vert \Sigma_{KK}^{-1}\left[\Sigma_{KK}-\hat{\Sigma}_{KK}\right]\hat{\Sigma}_{KK}^{-1}\right\Vert _{\infty}\nonumber \\
    & \leq & \sqrt{k}\left\Vert \Sigma_{KK}^{-1}\left[\Sigma_{KK}-\hat{\Sigma}_{KK}\right]\hat{\Sigma}_{KK}^{-1}\right\Vert _{2}\nonumber \\
    & \leq & \sqrt{k}\left\Vert \Sigma_{KK}^{-1}\right\Vert _{2}\left\Vert \Sigma_{KK}-\hat{\Sigma}_{KK}\right\Vert _{2}\left\Vert \hat{\Sigma}_{KK}^{-1}\right\Vert _{2}\nonumber \\
    & \leq & \frac{\sqrt{k}}{\lambda_{\min}(\Sigma_{KK})}\left\Vert \Sigma_{KK}-\hat{\Sigma}_{KK}\right\Vert _{2}\left\Vert \hat{\Sigma}_{KK}^{-1}\right\Vert _{2}.\label{eq:S5}
    \end{aligned}$$

To bound $\left\Vert \Sigma_{KK}-\hat{\Sigma}_{KK}\right\Vert _{2}$ in (\[eq:S5\]), we apply (\[eq:S3\]) with $t=\frac{\alpha^{2}}{\sqrt{k}}\sqrt{\frac{\log p}{n}}$ and obtain $$\left\Vert \hat{\Sigma}_{KK}-\Sigma_{KK}\right\Vert _{2}\leq\frac{\alpha^{2}}{\sqrt{k}}\sqrt{\frac{\log p}{n}},$$ with probability at least $1-c_{1}\exp\left(-c_{2}\frac{\log p}{k^{3}}\right)$, provided that $k^{-3}\frac{\log p}{n}\leq1$. To bound **$\left\Vert \hat{\Sigma}_{KK}^{-1}\right\Vert _{2}$** in (\[eq:S5\]), let us write $$\begin{aligned}
    \lambda_{\min}(\Sigma_{KK}) & = & \min_{||\mu^{'}||_{2}=1}\mu^{'T}\Sigma_{KK}\mu^{'}\nonumber \\
    & = & \min_{||\mu^{'}||_{2}=1}\left[\mu^{'T}\hat{\Sigma}_{KK}\mu^{'}+\mu^{'T}(\Sigma_{KK}-\hat{\Sigma}_{KK})\mu^{'}\right]\nonumber \\
    & \leq & \mu^{T}\hat{\Sigma}_{KK}\mu+\mu^{T}(\Sigma_{KK}-\hat{\Sigma}_{KK})\mu\label{eq:S6}
    \end{aligned}$$ where $\mu\in\mathbb{R}^{k}$ is a unit-norm minimal eigenvector of $\hat{\Sigma}_{KK}$. We then apply Lemma \[lem:A1\](b) with $t=\tilde{\alpha}^{2}\sqrt{\frac{\log p}{n}}$ to show $$\left|\mu^{T}\left(\Sigma_{KK}-\hat{\Sigma}_{KK}\right)\mu\right|\leq\tilde{\alpha}^{2}\sqrt{\frac{\log p}{n}}$$ with probability at least $1-c_{1}^{'}\exp\left(-c_{2}^{'}\log p\right)$, provided that $\sqrt{\frac{\log p}{n}}\leq1$. Therefore, $\lambda_{\min}(\Sigma_{KK})\leq\lambda_{\min}(\hat{\Sigma}_{KK})+\tilde{\alpha}^{2}\sqrt{\frac{\log p}{n}}$. As long as $\tilde{\alpha}^{2}\sqrt{\frac{\log p}{n}}\leq\frac{\lambda_{\min}(\Sigma_{KK})}{2}$, we have $$\lambda_{\min}(\hat{\Sigma}_{KK})\geq\frac{1}{2}\lambda_{\min}(\Sigma_{KK}),\label{eq:S7-1}$$ and consequently (\[eq:S8\]), $$\left\Vert \hat{\Sigma}_{KK}^{-1}\right\Vert _{2}\leq\frac{2}{\lambda_{\min}(\Sigma_{KK})}$$ with probability at least $1-c_{1}^{'}\exp\left(-c_{2}^{'}\log p\right)$.

Putting the pieces together, as long as $\frac{\alpha^{2}}{\lambda_{\min}(\Sigma_{KK})}\sqrt{\frac{\log p}{n}}\leq\frac{1}{12}$, $$\left\Vert \hat{\Sigma}_{KK}^{-1}-\Sigma_{KK}^{-1}\right\Vert _{\infty}\leq\frac{\sqrt{k}}{\lambda_{\min}(\Sigma_{KK})}\frac{\alpha^{2}}{\sqrt{k}}\sqrt{\frac{\log p}{n}}\frac{2}{\lambda_{\min}(\Sigma_{KK})}\leq\frac{1}{6\lambda_{\min}(\Sigma_{KK})}\label{eq:19}$$ with probability at least $1-c_{1}\exp\left(-c_{2}\frac{\log p}{k^{3}}\right)$.

\[lem:A3\] Let Assumption \[ass:random\_design\] hold. Suppose $$\left\Vert \mathbb{E}\left[X_{K^{c}}^{T}X_{K}\right]\left[\mathbb{E}(X_{K}^{T}X_{K})\right]^{-1}\right\Vert _{\infty}=1-\phi\label{eq:S1}$$ for some $\phi\in(0,\,1]$. If $k\geq1$ and $$\begin{aligned}
    \max\left\{ \frac{\phi}{12(1-\phi)k^{\frac{3}{2}}},\,\frac{\phi}{6k^{\frac{3}{2}}},\,\frac{\phi}{k}\right\} \sqrt{\frac{\log p}{n}} & \leq & \alpha^{2}\quad\textrm{if }\phi\in\left(0,\,1\right),\label{eq:7-2-1}\\
    \max\left\{ \frac{1}{6k^{\frac{3}{2}}},\,\frac{1}{k}\right\} \sqrt{\frac{\log p}{n}} & \leq & \alpha^{2}\quad\textrm{if }\phi=1,\label{eq:7-4-1}\\
    \max\left\{ 2\tilde{\alpha}^{2},\,12\alpha^{2},\,1\right\} \sqrt{\frac{\log p}{n}} & \leq & \lambda_{\min}(\Sigma_{KK}),\label{eq:7-3-1}
    \end{aligned}$$ then for some positive constant $b$ that only depends on $\phi$ and $\alpha$, we have $$\mathbb{P}\left[\left\Vert \frac{1}{n}X_{K^{c}}^{T}X_{K}\left(\frac{1}{n}X_{K}^{T}X_{K}\right)^{-1}\right\Vert _{\infty}\geq1-\frac{\phi}{2}\right]\leq c^{'}\exp\left(-b\left(\frac{\log p}{k^{3}}\right)\right).\label{eq:77}$$

Using the decomposition in @ravikumar2010, we have **$$\hat{\Sigma}_{K^{c}K}\hat{\Sigma}_{KK}^{-1}-\Sigma_{K^{c}K}\Sigma_{KK}^{-1}=R_{1}+R_{2}+R_{3},$$** where ****

$$\begin{aligned}
        R_{1} & = & \Sigma_{K^{c}K}\left[\hat{\Sigma}_{KK}^{-1}-\Sigma_{KK}^{-1}\right],\\
        R_{2} & = & \left[\hat{\Sigma}_{K^{c}K}-\Sigma_{K^{c}K}\right]\Sigma_{KK}^{-1},\\
        R_{3} & = & \left[\hat{\Sigma}_{K^{c}K}-\Sigma_{K^{c}K}\right]\left[\hat{\Sigma}_{KK}^{-1}-\Sigma_{KK}^{-1}\right].
    \end{aligned}$$

By (\[eq:S1\]), we have $\left\Vert \Sigma_{K^{c}K}\Sigma_{KK}^{-1}\right\Vert _{\infty}=1-\phi$. It suffices to show $\left\Vert R_{i}\right\Vert _{\infty}\leq\frac{\phi}{6}$ **** for $i=1,...,3$.

For $R_{1}$, note that $$R_{1}=-\Sigma_{K^{c}K}\Sigma_{KK}^{-1}[\hat{\Sigma}_{KK}-\Sigma_{KK}]\hat{\Sigma}_{KK}^{-1}.$$ Applying the facts $\left\Vert AB\right\Vert _{\infty}\leq\left\Vert A\right\Vert _{\infty}\left\Vert B\right\Vert _{\infty}$ and $\left\Vert A\right\Vert _{\infty}\leq\sqrt{a}\left\Vert A\right\Vert _{2}$ for any symmetric matrix $A\in\mathbb{R}^{a\times a}$, we can bound $R_{1}$ in the following fashion: $$\begin{aligned}
        \left\Vert R_{1}\right\Vert _{\infty} & \leq & \left\Vert \Sigma_{K^{c}K}\Sigma_{KK}^{-1}\right\Vert _{\infty}\left\Vert \hat{\Sigma}_{KK}-\Sigma_{KK}\right\Vert _{\infty}\left\Vert \hat{\Sigma}_{KK}^{-1}\right\Vert _{\infty}\\
        & \leq & (1-\phi)\left\Vert \hat{\Sigma}_{KK}-\Sigma_{KK}\right\Vert _{\infty}\sqrt{k}\left\Vert \hat{\Sigma}_{KK}^{-1}\right\Vert _{2},
    \end{aligned}$$ where the last inequality uses (\[eq:S1\]). If $\phi=1$, then $\left\Vert R_{1}\right\Vert _{\infty}=0$ so we may assume $\phi<1$ in the following. Bound (\[eq:S8\]) from the proof for Lemma \[lem:A2\] yields $$\left\Vert \hat{\Sigma}_{KK}^{-1}\right\Vert _{2}\leq\frac{2}{\lambda_{\min}(\Sigma_{KK})}$$ with probability at least $1-c_{1}\exp\left(-c_{2}\log p\right)$. Now, we apply bound (\[eq:S3\]) from Lemma \[lem:A2\] with $t=\frac{\phi}{12(1-\phi)}\sqrt{\frac{\log p}{kn}}$ and obtain $$\mathbb{P}\left[\left\Vert \hat{\Sigma}_{KK}-\Sigma_{KK}\right\Vert _{\infty}\geq\frac{\phi}{12(1-\phi)}\sqrt{\frac{\log p}{kn}}\right]\leq2\exp\left(-c\left(\frac{\phi^{2}\log p}{\alpha^{4}(1-\phi)^{2}k^{3}}\right)\right),$$ provided $\frac{\phi}{12(1-\phi)\alpha^{2}k}\sqrt{\frac{\log p}{kn}}\leq1$. Then, if $\sqrt{\frac{\log p}{n}}\leq\lambda_{\min}(\Sigma_{KK})$, we are guaranteed that $$\mathbb{P}\left[\left\Vert R_{1}\right\Vert _{\infty}\geq\frac{\phi}{6}\right]\leq2\exp\left(-c\left(\frac{\phi^{2}\log p}{\alpha^{4}(1-\phi)^{2}k^{3}}\right)\right)+c_{1}\exp\left(-c_{2}\log p\right).$$ For $R_{2}$, note that $$\begin{aligned}
        \left\Vert R_{2}\right\Vert _{\infty} & \leq & \sqrt{k}\left\Vert \Sigma_{KK}^{-1}\right\Vert _{2}\left\Vert \hat{\Sigma}_{K^{c}K}-\Sigma_{K^{c}K}\right\Vert _{\infty}\\
        & \leq & \frac{\sqrt{k}}{\lambda_{\min}(\Sigma_{KK})}\left\Vert \hat{\Sigma}_{K^{c}K}-\Sigma_{K^{c}K}\right\Vert _{\infty}.
    \end{aligned}$$ If $\frac{\phi}{6\alpha^{2}k}\sqrt{\frac{\log p}{kn}}\leq1$ and $\sqrt{\frac{\log p}{n}}\leq\lambda_{\min}(\Sigma_{KK})$, applying bound (\[eq:S2\]) from Lemma \[lem:A2\] with $t=\frac{\phi}{6}\sqrt{\frac{\log p}{kn}}$ yields $$\mathbb{P}\left[\left\Vert R_{2}\right\Vert _{\infty}\geq\frac{\phi}{6}\right]\leq2\exp\left(-c\left(\frac{\phi^{2}\log p}{\alpha^{4}k^{3}}\right)\right).$$ For $R_{3}$, applying (\[eq:S2\]) with $t=\phi\sqrt{\frac{\log p}{n}}$ to bound $\left\Vert \hat{\Sigma}_{K^{c}K}-\Sigma_{K^{c}K}\right\Vert _{\infty}$ and (\[eq:S4\]) to bound $\left\Vert \hat{\Sigma}_{KK}^{-1}-\Sigma_{KK}^{-1}\right\Vert _{\infty}$ yields

$$\mathbb{P}\left[\left\Vert R_{3}\right\Vert _{\infty}\geq\frac{\phi}{6}\right]\leq c^{'}\left[\exp\left(-c\left(\frac{\phi^{2}\log p}{\alpha^{4}k^{3}}\right)\right)+\exp\left(-c\left(\frac{\log p}{k^{3}}\right)\right)\right],$$ provided that $\frac{\phi}{\alpha^{2}k}\sqrt{\frac{\log p}{n}}\leq1$ and $\sqrt{\frac{\log p}{n}}\leq\lambda_{\min}(\Sigma_{KK})$.

Putting everything together, we conclude that $$\mathbb{P}\left[\left\Vert \hat{\Sigma}_{K^{c}K}\hat{\Sigma}_{KK}^{-1}\right\Vert _{\infty}\geq1-\frac{\phi}{2}\right]\leq c^{'}\exp\left(-b\left(\frac{\log p}{k^{3}}\right)\right)$$ for some positive constant $b$ that only depends on $\phi$ and $\alpha$.

\[lem:A4\] Let the assumptions in Lemmas \[lem:A2\] and \[lem:A3\] hold. Suppose $\theta^{*}$ is exactly sparse with at most $k$ non-zero coefficients and $K=\left\{ j:\,\theta_{j}^{*}\neq0\right\} \neq\emptyset$. If we choose $\lambda\geq\frac{c\alpha\sigma\left(2-\frac{\phi}{2}\right)}{\phi}\sqrt{\frac{\log p}{n}}$ for some sufficiently large universal constant $c>0$, $\left|\hat{\delta}_{K^{c}}\right|_{\infty}\leq1-\frac{\phi}{4}$ with probability at least $1-c_{1}\exp\left(-b\frac{\log p}{k^{3}}\right)$, where $b$ is some positive constant that only depends on $\phi$ and $\alpha$.

By construction, the subvectors $\hat{\theta}_{K}$, $\hat{\delta}_{K}$, and $\hat{\delta}_{K^{c}}$ satisfy the zero-subgradient condition in the PDW construction. With the fact that $\hat{\theta}_{K^{c}}=\theta_{K^{c}}^{*}=0_{p-k}$, we have $$\begin{aligned}
        \hat{\Sigma}_{KK}\left(\hat{\theta}_{K}-\theta_{K}^{*}\right)-\frac{1}{n}X_{K}^{T}\varepsilon+\lambda\hat{\delta}_{K} & = & 0_{k},\\
        \hat{\Sigma}_{K^{c}K}\left(\hat{\theta}_{K}-\theta_{K}^{*}\right)-\frac{1}{n}X_{K^{c}}^{T}\varepsilon+\lambda\hat{\delta}_{K^{c}} & = & 0_{p-k}.
    \end{aligned}$$ The equations above yields $$\begin{aligned}
        \hat{\delta}_{K^{c}} & = & -\frac{1}{\lambda}\hat{\Sigma}_{K^{c}K}\left(\hat{\theta}_{K}-\theta_{K}^{*}\right)+X_{K^{c}}^{T}\frac{\varepsilon}{n\lambda},\\
        \hat{\theta}_{K}-\theta_{K}^{*} & = & \hat{\Sigma}_{KK}^{-1}\frac{X_{K}^{T}\varepsilon}{n}-\lambda\hat{\Sigma}_{KK}^{-1}\hat{\delta}_{K},
    \end{aligned}$$ which yields $$\hat{\delta}_{K^{c}}=\left(\hat{\Sigma}_{K^{c}K}\hat{\Sigma}_{KK}^{-1}\right)\hat{\delta}_{K}+\left(X_{K^{c}}^{T}\frac{\varepsilon}{n\lambda}\right)-\left(\hat{\Sigma}_{K^{c}K}\hat{\Sigma}_{KK}^{-1}\right)X_{K}^{T}\frac{\varepsilon}{n\lambda}.$$ Using elementary inequalities and the fact that $\left|\hat{\delta}_{K}\right|_{\infty}\leq1$, we obtain $$\left|\hat{\delta}_{K^{c}}\right|_{\infty}\leq\left\Vert \hat{\Sigma}_{K^{c}K}\hat{\Sigma}_{KK}^{-1}\right\Vert _{\infty}+\left|X_{K^{c}}^{T}\frac{\varepsilon}{n\lambda}\right|_{\infty}+\left\Vert \hat{\Sigma}_{K^{c}K}\hat{\Sigma}_{KK}^{-1}\right\Vert _{\infty}\left|X_{K}^{T}\frac{\varepsilon}{n\lambda}\right|_{\infty}.$$

By Lemma \[lem:A3\], $\left\Vert \hat{\Sigma}_{K^{c}K}\hat{\Sigma}_{KK}^{-1}\right\Vert _{\infty}\leq1-\frac{\phi}{2}$ with probability at least $1-c^{'}\exp\left(-\frac{b\log p}{k^{3}}\right)$; as a result, $$\begin{aligned}
        \left|\hat{\delta}_{K^{c}}\right|_{\infty} & \leq & 1-\frac{\phi}{2}+\left|X_{K^{c}}^{T}\frac{\varepsilon}{n\lambda}\right|_{\infty}+\left\Vert \hat{\Sigma}_{K^{c}K}\hat{\Sigma}_{KK}^{-1}\right\Vert _{\infty}\left|X_{K}^{T}\frac{\varepsilon}{n\lambda}\right|_{\infty}\\
        & \leq & 1-\frac{\phi}{2}+\left(2-\frac{\phi}{2}\right)\left|\hat{X}^{T}\frac{\varepsilon}{n\lambda}\right|_{\infty}.
    \end{aligned}$$ It remains to show that $\left(2-\frac{\phi}{2}\right)\left|X^{T}\frac{\varepsilon}{n\lambda}\right|_{\infty}\leq\frac{\phi}{4}$ with high probability. This result holds if $\lambda\geq\frac{4\left(2-\frac{\phi}{2}\right)}{\phi}\left|X^{T}\frac{\varepsilon}{n}\right|_{\infty}$. In particular, Lemma \[lem:A1\](a) and a union bound imply that $$\mathbb{P}\left(\left|\frac{X^{T}\varepsilon}{n}\right|_{\infty}\geq t\right)\leq2\exp\left(\frac{-nt^{2}}{c_{0}\sigma^{2}\alpha^{2}}+\log p\right).$$ Thus, under the choice of $\lambda$ in Lemma \[lem:A4\], we have $\left|\hat{\delta}_{K^{c}}\right|_{\infty}\leq1-\frac{\phi}{4}$ with probability at least $1-c_{1}\exp\left(-b\frac{\log p}{k^{3}}\right)$.

[^1]: Department of Economics, University of California, San Diego, 9500 Gilman Dr. La Jolla, CA 92093. Email: kwuthrich@ucsd.edu

[^2]: Department of Economics, University of California, San Diego, 9500 Gilman Dr. La Jolla, CA 92093. Email: yiz012@ucsd.edu.

[^3]: Alphabetical ordering; both authors contributed equally to this work. We would like to thank Stéphane Bonhomme, Gordon Dahl, Graham Elliott, Michael Jansson, Ulrich Müller, Andres Santos, Azeem Shaikh, Aman Ullah, Jeffrey Wooldridge, and seminar participants for their comments. We are especially grateful to Yixiao Sun for providing extensive feedback on an earlier draft. This paper was previously circulated as “Behavior of Lasso and Lasso-based inference under limited variability” and “Omitted variable bias of Lasso-based inference methods under limited variability: A finite sample analysis”. Zhu acknowledges a start-up fund from the Department of Economics at UCSD and the Department of Statistics and the Department of Computer Science at Purdue University, West Lafayette.

[^4]: A natural idea to avoid OVBs due to the Lasso not selecting relevant controls is to choose a regularization parameter smaller than the recommended ones [e.g., @belloni2014inference]. However, as we show in simulations, this idea does not work in general and can lead to substantial biases.

[^5]: The existing theoretical framework for the Lasso-based inference methods makes it difficult to derive an informative $\text{constant}$. To our knowledge, the literature provides no such derivations.

[^6]: Note that the existing Lasso theory requires the regularization parameter to exceed a certain threshold, which depends on the standard deviations of the noise and the covariates.

[^7]: Such a comparison is useful as the standard deviation in @belloni2014inference is the basis for their recommended inference procedures. We would also like to compare the OVBs to the conditional or unconditional finite sample standard deviation. However, this would require exact formulas of various selection probabilities (more than just bounds on the probabilities), which are impossible to derive for the Lasso in general.

[^8]: Many applications of Lasso-based inference procedures feature such specifications [e.g., @belloni2014high; @belloni2014inference; @chen2015can; @decker2016health; @fremstad2017does; @knaus2018heterogenous; @jones2018what; @schmitz2017informal].

[^9]: In our theoretical results, however, $p$ is allowed to exceed $n$.

[^10]: Note that traditional robust standard errors are inconsistent in settings with many controls [@cattaneo2018inference].

[^11]: Specifically, we set $\lambda=2\sigma\sqrt{\frac{2b(1+\tau)\log p}{n}}$, assuming that $\sigma$ is known. In practice, we first normalize $X_{i}$ such that $b=1$, run Lasso using $\lambda=2\sigma\sqrt{\frac{2(1+\tau)\log p}{n}}$, and then rescale the coefficients.

[^12]: For all four values of $\sigma_x$, none of the controls with zero coefficients get selected with high probability.

[^13]: Note that exact formulas of these mixture probabilities cannot be derived.

[^14]: For both values of $\sigma_x$, none of the controls with zero coefficients get selected with high probability.

[^15]: This approach is based on the following modified Lasso program: $$\hat{\theta}\in\arg\min_{\theta\in\mathbb{R}^{p}}\frac{1}{n}\sum_{i=1}^n\left(Y_i-X_i\theta \right)^2+\frac{\lambda}{n}\sum_{j=1}^p|\hat{l}_j\theta_j |
    \label{eq:general_lasso}$$ where $(\hat{l}_1,\dots,\hat{l}_p)$ are penalty loadings obtained using the iterative post Lasso-based algorithm developed in @belloni2012sparse. Our implementation is based on the Matlab code provided on the authors’ webpage: <https://voices.uchicago.edu/christianhansen/code-and-data/>. We set $\lambda=2c\sqrt{n}\Phi^{-1}(1-\varsigma/(2p))$, where $c=1.1$ and $\varsigma=0.1/\log n$ as recommended by @belloni2014inference [@belloni2017program]. Under homoscedasticity and with nearly orthogonal $X$, this regularization choice has a similar magnitude as the one in @bickel2009simultaneous.\[footnote\_bcch\]

[^16]: \[footnote:over-selection\] A similar phenomenon arises with $0.5\lambda_{\text{BCCH}}$. This choice works well when $\alpha^\ast=0$, but yields biases when $\alpha^\ast\ne 0$. We found that, under our DGPs, this is related to the fact that when $\alpha^\ast\ne 0$, and differ in terms of the underlying coefficients and the noise level, which leads to differences in the (over-)selection behavior of the Lasso.

[^17]: Here focus on the intention to treat effect of 401(k) eligibility on assets as in @Poterbaetal1994 [@Poterbaetal1995; @Poterbaetal1998] and @Benjamin2003. Some studies use 401(k) eligibility as an instrument for 401(k) participation [e.g., @CH2004; @belloni2017program].

[^18]: @fryerlevitt2013 also analyze test scores at the age of four years as well as eight months.

[^19]: With interviewer fixed effects and based on the sample of all children, we were able to exactly replicate the results in Table 3 of @fryerlevitt2013.

[^20]: The normality of $\varepsilon_{i}$ can be relaxed without changing the essence of our results.

[^21]: Here (and similarly in the rest of propositions), we implicitly assume $p$ is sufficiently large such that $1-k\exp\left(\frac{-b^{2}\left(1+\tau\right)\log p}{4\phi^{2}}\right)-\frac{2}{p^{\tau}}>0$. Indeed, probabilities in such a form are often referred to as the “high probability” guarantees in the literature of (nonasymptotic) high dimensional statistics concerning large $p$ and small enough $k$. The event $\mathcal{M}$ is the intersection of $\left\{ \hat{I}_{1}=\hat{I}_{2}=\emptyset\right\}$ and an additional event; $\left\{ \hat{I}_{1}=\hat{I}_{2}=\emptyset\right\}$ occurs with probability at least $1-\frac{2}{p^{\tau}}$ and the additional event occurs with probability at least $1-k\exp\left(\frac{-b^{2}\left(1+\tau\right)\log p}{4\phi^{2}}\right)$. The additional event is needed in our analyses for technical reasons. See (\[eq:M\_event\]) of Appendix \[sec:appendix\_post\_double\_lasso\] for the definition of $\mathcal{M}$.

[^22]: We thank Ulrich Müller for suggesting this comparison.

[^23]: In general, $\lambda_{1}\asymp\phi^{-1}\left(\sigma_{\eta}+\left|\alpha^{*}\right|\sigma_{v}\right)\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}$ and the iterative algorithm for choosing $\lambda_{1}$ in @belloni2014inference described in Footnote \[footnote\_bcch\] achieves this scaling. Under the conditions on $\left(\beta_{K}^{*},\,\gamma_{K}^{*},\,\alpha^{*}\right)$ in Proposition \[prop:bias\_post\_double\_selection\], this scaling is equivalent to $\phi^{-1}\sigma_{\eta}\sqrt{\frac{s}{n}}\sqrt{\frac{\log p}{n}}$.

[^24]: In fact, when the parameter space is unrestricted, OLS-based inference exhibits desirable optimality properties [e.g., @armstrong2016 Section 4.1].

[^25]: For a convex function $f:\,\mathbb{R}^{p}\mapsto\mathbb{R}$, $\delta\in\mathbb{R}^{p}$ is a subgradient at $\theta$, namely $\delta\in\partial f(\theta)$, if $f(\theta+\triangle)\geq f(\theta)+\left\langle \delta,\,\triangle\right\rangle $ for all $\triangle\in\mathbb{R}^{p}$.

[^26]: Because $v$ and $\eta$ are independent of each other, the bound $\mathbb{P}\left(E\cap\mathcal{E}_{t^{*}}\right)\geq1-k\exp\left(\frac{-b^{2}\left(1+\tau\right)\log p}{4\phi^{2}}\right)-\frac{2}{p^{\tau}}$ can be further sharpened to $\mathbb{P}\left(E\cap\mathcal{E}_{t^{*}}\right)\geq\left(1-\frac{1}{p^{\tau}}\right)^{2}-k\exp\left(\frac{-b^{2}\left(1+\tau\right)\log p}{4\phi^{2}}\right)$.

[^27]: We found that one of the reasons for the relatively poor performance of debiased Lasso is that $D$ is highly correlated with the relevant controls. Unreported simulation results show that debiased Lasso exhibits a better performance when $(D,X)$ exhibit a Toeplitz dependence structure as in the simulations reported by @vandergeer2014asymptotically.

[^28]: This model and the multiplicative specification of heteroscedasticity are as in the simulations of @belloni2014inference.
