{"action":"created","issue":{"url":"https://api.github.com/repos/topepo/caret/issues/212","labels_url":"https://api.github.com/repos/topepo/caret/issues/212/labels{/name}","comments_url":"https://api.github.com/repos/topepo/caret/issues/212/comments","events_url":"https://api.github.com/repos/topepo/caret/issues/212/events","html_url":"https://github.com/topepo/caret/issues/212","id":99435550,"number":212,"title":"Inconsistent results depending on models used","user":{"login":"lucasxteixeira","id":5638293,"avatar_url":"https://avatars.githubusercontent.com/u/5638293?v=3","gravatar_id":"","url":"https://api.github.com/users/lucasxteixeira","html_url":"https://github.com/lucasxteixeira","followers_url":"https://api.github.com/users/lucasxteixeira/followers","following_url":"https://api.github.com/users/lucasxteixeira/following{/other_user}","gists_url":"https://api.github.com/users/lucasxteixeira/gists{/gist_id}","starred_url":"https://api.github.com/users/lucasxteixeira/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lucasxteixeira/subscriptions","organizations_url":"https://api.github.com/users/lucasxteixeira/orgs","repos_url":"https://api.github.com/users/lucasxteixeira/repos","events_url":"https://api.github.com/users/lucasxteixeira/events{/privacy}","received_events_url":"https://api.github.com/users/lucasxteixeira/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-08-06T13:35:24Z","updated_at":"2015-08-08T19:30:23Z","closed_at":"2015-08-06T14:04:10Z","body":"The snippet below using gbm produces an AUC of 0.9155\r\n\r\n```{r}\r\nlibrary(caret)\r\nlibrary(mlbench)\r\nlibrary(pROC)\r\ndata(Sonar)\r\nset.seed(1234)\r\n\r\ninTrain <- caret::createDataPartition(y = Sonar[, \"Class\"], p = .5, list = FALSE)\r\ndataTrain <- Sonar[inTrain, ]\r\ndataTest <- Sonar[-inTrain, ]\r\n\r\ntrControl <- caret::trainControl(method = \"repeatedcv\", number = 5, repeats = 3,\r\n    classProbs = TRUE, savePredictions = TRUE, summaryFunction = caret::twoClassSummary,\r\n    index = caret::createMultiFolds(dataTrain$Class, k = 5, times = 3)\r\n)\r\n\r\ngbmModel <- caret::train(\r\n    Class ~ . , \r\n    data = dataTrain,\r\n    method = \"gbm\",\r\n    trControl = trControl,\r\n    preProcess = c(\"center\", \"scale\"),\r\n    metric = \"ROC\"\r\n)\r\n\r\ngbmProbs <- caret::predict.train(gbmModel, newdata = dataTest, type = \"prob\")\r\nplot( pROC::roc(response = dataTest[,\"Class\"], predictor = gbmProbs[,\"M\"]), print.auc=TRUE )\r\n```\r\n\r\nHowever, when I train more models the AUC changes.\r\nGiven other snippet below, gbm AUC is now 0.9102\r\n\r\n```{r}\r\nlibrary(caret)\r\nlibrary(mlbench)\r\nlibrary(pROC)\r\ndata(Sonar)\r\nset.seed(1234)\r\n\r\ninTrain <- caret::createDataPartition(y = Sonar[, \"Class\"], p = .5, list = FALSE)\r\ndataTrain <- Sonar[inTrain, ]\r\ndataTest <- Sonar[-inTrain, ]\r\n\r\ntrControl <- caret::trainControl(method = \"repeatedcv\", number = 5, repeats = 3,\r\n    classProbs = TRUE, savePredictions = TRUE, summaryFunction = caret::twoClassSummary,\r\n    index = caret::createMultiFolds(dataTrain$Class, k = 5, times = 3)\r\n)\r\n\r\nbagFDAModel <- caret::train(\r\n    Class ~ . , \r\n    data = dataTrain,\r\n    method = \"bagFDA\",\r\n    trControl = trControl,\r\n    preProcess = c(\"center\", \"scale\"),\r\n    metric = \"ROC\"\r\n)\r\n\r\nbagFDAProbs <- caret::predict.train(bagFDAModel, newdata = dataTest, type = \"prob\")\r\nplot( pROC::roc(response = dataTest[,\"Class\"], predictor = bagFDAProbs[,\"M\"]), print.auc=TRUE )\r\n\r\ngbmModel <- caret::train(\r\n    Class ~ . , \r\n    data = dataTrain,\r\n    method = \"gbm\",\r\n    trControl = trControl,\r\n    preProcess = c(\"center\", \"scale\"),\r\n    metric = \"ROC\"\r\n)\r\n\r\ngbmProbs <- caret::predict.train(gbmModel, newdata = dataTest, type = \"prob\")\r\nplot( pROC::roc(response = dataTest[,\"Class\"], predictor = gbmProbs[,\"M\"]), print.auc=TRUE )\r\n```\r\n\r\nThat is just an example, this also happens with different combinations of models."},"comment":{"url":"https://api.github.com/repos/topepo/caret/issues/comments/129040056","html_url":"https://github.com/topepo/caret/issues/212#issuecomment-129040056","issue_url":"https://api.github.com/repos/topepo/caret/issues/212","id":129040056,"user":{"login":"MikeBadescu","id":6641575,"avatar_url":"https://avatars.githubusercontent.com/u/6641575?v=3","gravatar_id":"","url":"https://api.github.com/users/MikeBadescu","html_url":"https://github.com/MikeBadescu","followers_url":"https://api.github.com/users/MikeBadescu/followers","following_url":"https://api.github.com/users/MikeBadescu/following{/other_user}","gists_url":"https://api.github.com/users/MikeBadescu/gists{/gist_id}","starred_url":"https://api.github.com/users/MikeBadescu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MikeBadescu/subscriptions","organizations_url":"https://api.github.com/users/MikeBadescu/orgs","repos_url":"https://api.github.com/users/MikeBadescu/repos","events_url":"https://api.github.com/users/MikeBadescu/events{/privacy}","received_events_url":"https://api.github.com/users/MikeBadescu/received_events","type":"User","site_admin":false},"created_at":"2015-08-08T19:30:23Z","updated_at":"2015-08-08T19:30:23Z","body":"There is the a problem with `oob` for `randomForest`. Here a code sample based on a [stackoverflow answer](http://stackoverflow.com/a/21988897):\r\n\r\n```r\r\nlibrary(doParallel); library(caret)\r\n\r\n#create a list of seed, here change the seed for each resampling\r\nset.seed(123)\r\nseeds <- vector(mode = \"list\", length = 10 * 1 + 1)#length is = (n_repeats*nresampling)+1\r\nfor(i in 1:10) seeds[[i]]<- sample.int(n=1000, 3) #(3 is the number of tuning parameter, mtry for rf, here equal to ncol(iris)-2)\r\n\r\nseeds[[11]]<-sample.int(1000, 1)#for the last model\r\n\r\n#control list\r\nset.seed(123)\r\nmyControl <- trainControl(method='oob', number = 10, repeats = 1,\r\n                          seeds=seeds, index=createFolds(iris$Species, 10))\r\n\r\n#run model in parallel\r\ncl <- makeCluster(detectCores())\r\nregisterDoParallel(cl)\r\n\r\nset.seed(123)\r\nprint(model1 <- train(Species~., iris, method='rf', trControl=myControl))\r\n\r\nset.seed(123)\r\nprint(model2 <- train(Species~., iris, method='rf', trControl=myControl))\r\n\r\nstopCluster(cl)\r\n\r\n#compare\r\nprint(all.equal(predict(model1, type='prob'), predict(model2, type='prob')))\r\n```\r\n\r\nRun it several times to notice the difference. Multiple `set.seed` and use of partitions are not enough to guarantee reproducibility. Maybe incorporating [doRNG](https://cran.r-project.org/web/packages/doRNG/index.html) might help. "}}