R caret: leave subject out cross validation with data subset for training?
<p>I want to perform leave subject out cross validation with R caret (cf. <a href="https://stats.stackexchange.com/questions/109340/leave-one-subject-out-cross-validation-in-caret">this example</a>) but only use a subset of the data in training for creating CV models. Still, the left out CV partition should be used as a whole, as I need to test on all data of a left out subject (no matter if it's millions of samples that cannot be used in training due to computational restrictions). </p>

<p>I've created a minimal 2 class classification example using the <code>subset</code> and <code>index</code> parameters of <code>caret::train</code> and <code>caret::trainControl</code> to achieve this. From my observation this should solve the problem, but I have a hard time actually <em>ensuring that the evaluation is still done in a leave-subject-out way</em>. Maybe someone with experience in this task could shed some light on this:</p>

<pre><code>library(plyr)
library(caret)
library(pROC)
library(ggplot2)

# with diamonds we want to predict cut and look at results for different colors = subjects
d &lt;- diamonds
d &lt;- d[d$cut %in% c('Premium', 'Ideal'),] # make a 2 class problem
d$cut &lt;- factor(d$cut)
indexes_data &lt;- c(1,5,6,8:10)
indexes_labels &lt;- 2 

# population independent CV indexes for trainControl
index &lt;- llply(unique(d[,3]), function(cls) c(which(d[,3]!=cls)))
names(index) &lt;- paste0('sub_', unique(d[,3]))
str(index) # indexes used for training models with CV = OK

m3 &lt;- train(x = d[,indexes_data], 
            y = d[,indexes_labels], 
            method = 'glm', 
            metric = 'ROC',
            subset = sample(nrow(d), 5000), # does this subset the data used for training and obtaining models, but not the left out partition used for estimating CV performance?
            trControl = trainControl(returnResamp = 'final', 
                                    savePredictions = T,
                                    classProbs = T, 
                                    summaryFunction = twoClassSummary, 
                                    index = index))
str(m3$resample) # all samples used once = OK

# performance over all subjects
myRoc &lt;- roc(predictor = m3$pred[,3], response = m3$pred$obs)
</code></pre>

<p>plot(myRoc, main = 'all')</p>

<h1>performance for individual subjects</h1>

<p>l_ply(unique(m3$pred$Resample), .fun = function(cls) {
    pred_sub &lt;- m3$pred[m3$pred$Resample==cls,]
    myRoc &lt;- roc(predictor = pred_sub[,3], response = pred_sub$obs)
        plot(myRoc, main = cls)
    } )</p>

<p>Thanks for your time!</p>