I have a dataset that I divide into two equal partitions A and B.

I estimate a regression model on partition A.

I want to calculate the cross-validated $R^2$ when predicting the values in partition B.

I would like to know if the following approach is correct and also what other ways there could be:

#generate data:

data <- replicate(10, rnorm(100))
data <- as.data.frame(data)

#divide into training and test set:

train <- data[1:50,]
test <- data[51:100,]

#fit model and get predictions for unseen data:

model <- lm(train[,1] ~., data = train)
predictions <- predict(model, test)

#obtain cross-validated R squared:


1 Answer 1


That is incorrect because it allows for recalibration of predictions with a new overall slope and intercept. Use this formula after freezing all coefficients: 1 - (sum of squared errors) / (sum of squares total). The denominator is $(n-1)\times$ the observed variance of $Y$ in the holdout sample. When you do it correctly you can get negative $R^2$ in some holdout samples when the real $R^2$ is low.

  • $\begingroup$ Thank you for your help! So the correct formula is 1-(sum((data[,1] - predictions)^2) / (n-1) * var(test[,1]) ? $\endgroup$ Dec 12, 2015 at 16:21
  • $\begingroup$ p.s. what do you mean by freezing all coefficients? $\endgroup$ Dec 12, 2015 at 16:22
  • 1
    $\begingroup$ (+1) "Freezing" means you don't get to refit your model on the test set. The test set is a proxy for any additional data to which you would like to apply the model you have--which at this stage is the one fit only on the training data. (I have recently discovered, by observing what my own students do, that the misconception reflected in this question is a common one. I found that displaying a plot of predicted vs. actual values for the test set and discussing its construction could alleviate this confusion.) $\endgroup$
    – whuber
    Dec 12, 2015 at 16:26
  • $\begingroup$ @whuber: I don't get it. Why does my approach allow for recalibration of predictions? I am estimating the model on one partition and using it to predict unseen data. Can you please clarify which part of my approach is incorrect? Thank you! $\endgroup$ Dec 12, 2015 at 17:10
  • 2
    $\begingroup$ It is very incorrect. Say your predictions are off by a multiple of 2 and an additional increment of 10. The correlation coefficient will not penalize for that, internally allowing for a new slope and intercept to be estimated on top of the predictions. A correlation is a relative measure whereas sum of squared errors is absolute. $\endgroup$ Dec 13, 2015 at 21:10

Your Answer

