{"action":"created","issue":{"url":"https://api.github.com/repos/lme4/lme4/issues/231","labels_url":"https://api.github.com/repos/lme4/lme4/issues/231/labels{/name}","comments_url":"https://api.github.com/repos/lme4/lme4/issues/231/comments","events_url":"https://api.github.com/repos/lme4/lme4/issues/231/events","html_url":"https://github.com/lme4/lme4/issues/231","id":40443955,"number":231,"title":"\"bootstrap runs failed\" in bootMer within confint.merMod","user":{"login":"pcdjohnson","id":5443341,"avatar_url":"https://avatars.githubusercontent.com/u/5443341?v=3","gravatar_id":"","url":"https://api.github.com/users/pcdjohnson","html_url":"https://github.com/pcdjohnson","followers_url":"https://api.github.com/users/pcdjohnson/followers","following_url":"https://api.github.com/users/pcdjohnson/following{/other_user}","gists_url":"https://api.github.com/users/pcdjohnson/gists{/gist_id}","starred_url":"https://api.github.com/users/pcdjohnson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcdjohnson/subscriptions","organizations_url":"https://api.github.com/users/pcdjohnson/orgs","repos_url":"https://api.github.com/users/pcdjohnson/repos","events_url":"https://api.github.com/users/pcdjohnson/events{/privacy}","received_events_url":"https://api.github.com/users/pcdjohnson/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2014-08-17T21:30:31Z","updated_at":"2015-03-29T18:03:27Z","closed_at":null,"body":"I’ve been using confint.merMod to get 95% CIs by parametric bootstrapping, and have been getting the warnings of the type \"In bootMer(object, bootFun, nsim = nsim, ...) : some bootstrap runs failed (30/100)”. The function still returns CIs using the bootstrap samples that didn’t fail, but I don’t think these CIs are valid as the failures are very unlikely to be a random sample. Here’s an example using the grouseticks data. I chose this data set because it’s a classic example data set so presumably isn’t pathological, but I’ve been encountering the same problem with other apparently healthy glmer fits.\r\n\r\n> library(lme4)\r\n> full_mod1  <- glmer(TICKS ~ YEAR+scale(HEIGHT)+(1|BROOD)+(1|INDEX)+(1|LOCATION), family=\"poisson\", data=grouseticks)\r\n> set.seed(12345)\r\n> confint(full_mod1, method = \"boot\", boot.type = \"basic\", nsim = 100, parallel = \"multicore\", ncpus = 8)\r\nComputing bootstrap confidence intervals ...\r\n                                2.5 %     97.5 %\r\nsd_(Intercept)|INDEX     0.4624700492  0.6550336\r\nsd_(Intercept)|BROOD     0.5740796527  1.0482547\r\nsd_(Intercept)|LOCATION  0.2497343230  1.0574176\r\n(Intercept)             -0.0002611865  0.7208569\r\nYEAR96                   0.7770837606  1.5857202\r\nYEAR97                  -1.4648325648 -0.4484355\r\nscale(HEIGHT)           -1.1335522182 -0.5879858\r\nWarning message:\r\nIn bootMer(object, bootFun, nsim = nsim, ...) :\r\n  some bootstrap runs failed (30/100)\r\n\r\n(NB I scaled height to prevent a warning but it makes no difference to the likelihood.)\r\n\r\nI tried to replicate the problem using bootMer directly to produce fixed effect estimates from 10 samples:\r\n\r\n> bootMer(full_mod1, fixef, nsim = 10)$t\r\n      (Intercept)    YEAR96     YEAR97 scale(HEIGHT)\r\n [1,]   0.6387534 0.7467317 -1.1291369    -0.8741711\r\n [2,]          NA        NA         NA            NA\r\n [3,]   0.1920928 1.3666922 -0.5821328    -0.6742498\r\n [4,]          NA        NA         NA            NA\r\n [5,]          NA        NA         NA            NA\r\n [6,]   0.5312426 0.9639612 -1.1041643    -0.7550492\r\n [7,]          NA        NA         NA            NA\r\n [8,]   0.2328530 1.1016917 -0.8010541    -0.8480263\r\n [9,]   0.2685594 1.2806690 -0.7452719    -0.8302675\r\n[10,]   0.1726928 1.1423894 -0.8989542    -1.0344128\r\nWarning message:\r\nIn bootMer(full_mod1, fixef, nsim = 10) : some bootstrap runs failed (4/10)\r\n\r\n…and get the same problem, not surprisingly. Here’s a manual version of what (I believe) bootMer is doing:\r\n\r\n> simTICKS.tab <- simulate(full_mod1, nsim=10)\r\n> t(apply(simTICKS.tab, 2, function(simTICKS) fixef(glmer(simTICKS ~ YEAR+scale(HEIGHT)+(1|BROOD)+(1|INDEX)+(1|LOCATION), family=\"poisson\", data=grouseticks))))\r\n       (Intercept)    YEAR96     YEAR97 scale(HEIGHT)\r\nsim_1    0.1793276 1.2265976 -0.5753168    -0.6435871\r\nsim_2    0.3918824 1.4583529 -0.8958396    -0.8513233\r\nsim_3    0.2565916 1.1230342 -0.9369728    -1.0817019\r\nsim_4    0.3507285 1.1865904 -1.0654895    -0.9193403\r\nsim_5    0.5688213 0.7291811 -0.9734654    -0.8575988\r\nsim_6    0.6160919 1.0552260 -1.5139770    -1.0050926\r\nsim_7    0.1594954 1.1342809 -1.0624134    -0.8107930\r\nsim_8    0.7956874 0.8290606 -1.2491060    -1.0024299\r\nsim_9    0.3081545 1.2677562 -1.0398100    -0.9458849\r\nsim_10   0.4325345 0.9390844 -0.9241206    -0.7341589\r\n\r\n…no errors, not even a warning. The discrepancy between bootMer and the DIY version above appears to arise from the bootMer fitting function, refit, being less tolerant of warning signs (non-convergence?) than glmer, which can be seen by putting refit into the DIY bootstrap:\r\n\r\n> t(apply(simTICKS.tab, 2, function(simTICKS) fixef(refit(full_mod1, simTICKS))))\r\nWarning messages:\r\n1: In pwrssUpdate(pp, resp, tolPwrss, GQmat, compDev, fac, verbose) :\r\n  Cholmod warning 'not positive definite' at file:../Cholesky/t_cholmod_rowfac.c, line 431\r\n2: In pwrssUpdate(pp, resp, tolPwrss, GQmat, compDev, fac, verbose) :\r\n  Cholmod warning 'not positive definite' at file:../Cholesky/t_cholmod_rowfac.c, line 431\r\nError in t(apply(simTICKS.tab, 2, function(simTICKS) fixef(refit(full_mod1,  : \r\n  error in evaluating the argument 'x' in selecting a method for function 't': Error in t(apply(simTICKS.tab, 2, function(simTICKS) fixef(refit(full_mod1,  : \r\n  (maxstephalfit) PIRLS step-halvings failed to reduce deviance in pwrssUpdate\r\n\r\nMy questions are: \r\n\r\nShould I trust the error- and warning-free DIY bootstrap results? I would say “yes”, on the (slightly vague) basis that glmer produced no warnings, and that the original model fits well to what is a pretty information-rich data set. So is bootMer being oversensitive? I’m aware that there have been some teething problems with the optimisers and convergence checks in lme4 1.0+ — are these ongoing? \r\n\r\n> sessionInfo()\r\nR version 3.1.1 (2014-07-10)\r\nPlatform: x86_64-apple-darwin13.1.0 (64-bit)\r\n\r\nlocale:\r\n[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8\r\n\r\nattached base packages:\r\n[1] parallel  stats     graphics  grDevices utils     datasets  methods   base     \r\n\r\nother attached packages:\r\n[1] lme4_1.1-7   Rcpp_0.11.2  Matrix_1.1-4\r\n\r\nloaded via a namespace (and not attached):\r\n[1] boot_1.3-11     grid_3.1.1      lattice_0.20-29 MASS_7.3-33     minqa_1.2.3     nlme_3.1-117    nloptr_1.0.0    splines_3.1.1   tools_3.1.1"},"comment":{"url":"https://api.github.com/repos/lme4/lme4/issues/comments/87450760","html_url":"https://github.com/lme4/lme4/issues/231#issuecomment-87450760","issue_url":"https://api.github.com/repos/lme4/lme4/issues/231","id":87450760,"user":{"login":"bbolker","id":78918,"avatar_url":"https://avatars.githubusercontent.com/u/78918?v=3","gravatar_id":"","url":"https://api.github.com/users/bbolker","html_url":"https://github.com/bbolker","followers_url":"https://api.github.com/users/bbolker/followers","following_url":"https://api.github.com/users/bbolker/following{/other_user}","gists_url":"https://api.github.com/users/bbolker/gists{/gist_id}","starred_url":"https://api.github.com/users/bbolker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bbolker/subscriptions","organizations_url":"https://api.github.com/users/bbolker/orgs","repos_url":"https://api.github.com/users/bbolker/repos","events_url":"https://api.github.com/users/bbolker/events{/privacy}","received_events_url":"https://api.github.com/users/bbolker/received_events","type":"User","site_admin":false},"created_at":"2015-03-29T18:03:27Z","updated_at":"2015-03-29T18:03:27Z","body":"Have fixed that problem, but it doesn't fix the larger `reset()` problem.  I think we still need to go through and see exactly how `lp0` is being reset (and figure out how it ought to be).  Right now we have `baseOffset <- object@resp$offset; ...; lp0=object@resp$eta - baseOffset`.  I don't know whether one or both of these ought to be deep copies, nor whether they are being set correctly, nor whether this will actually work (given that `eta` and `offset` are both in different conditions at the beginning and end of each stage of the fitting process ...)\r\n"}}