Relation between PLS and OLS regression in terms of the eigenvalue
  distribution of the regressor covariance matrix
David del Val José R. Berrendero Alberto Suárez
Partial least squares (PLS) is a dimensionality reduction technique
introduced in the field of chemometrics and successfully employed in many other
areas. The PLS components are obtained by maximizing the covariance between
linear combinations of the regressors and of the target variables. In this
work, we focus on its application to scalar regression problems. PLS regression
consists in finding the least squares predictor that is a linear combination of
a subset of the PLS components. Alternatively, PLS regression can be formulated
as a least squares problem restricted to a Krylov subspace. This equivalent
formulation is employed to analyze the distance between
${\hat{\boldsymbol\beta}\;}_{\mathrm{PLS}}^{\scriptscriptstyle {(L)}}$, the PLS
estimator of the vector of coefficients of the linear regression model based on
$L$ PLS components, and $\hat{\boldsymbol \beta}_{\mathrm{OLS}}$, the one
obtained by ordinary least squares (OLS), as a function of $L$. Specifically,
${\hat{\boldsymbol\beta}\;}_{\mathrm{PLS}}^{\scriptscriptstyle {(L)}}$ is the
vector of coefficients in the aforementioned Krylov subspace that is closest to
$\hat{\boldsymbol \beta}_{\mathrm{OLS}}$ in terms of the Mahalanobis distance
with respect to the covariance matrix of the OLS estimate. We provide a bound
on this distance that depends only on the distribution of the eigenvalues of
the regressor covariance matrix. Numerical examples on synthetic and real-world
data are used to illustrate how the distance between
${\hat{\boldsymbol\beta}\;}_{\mathrm{PLS}}^{\scriptscriptstyle {(L)}}$ and
$\hat{\boldsymbol \beta}_{\mathrm{OLS}}$ depends on the number of clusters in
which the eigenvalues of the regressor covariance matrix are grouped.