{"action":"created","issue":{"url":"https://api.github.com/repos/pandegroup/openmm/issues/1108","labels_url":"https://api.github.com/repos/pandegroup/openmm/issues/1108/labels{/name}","comments_url":"https://api.github.com/repos/pandegroup/openmm/issues/1108/comments","events_url":"https://api.github.com/repos/pandegroup/openmm/issues/1108/events","html_url":"https://github.com/pandegroup/openmm/issues/1108","id":103145054,"number":1108,"title":"Energy discrepancies between platforms on large systems","user":{"login":"jchodera","id":3656088,"avatar_url":"https://avatars.githubusercontent.com/u/3656088?v=3","gravatar_id":"","url":"https://api.github.com/users/jchodera","html_url":"https://github.com/jchodera","followers_url":"https://api.github.com/users/jchodera/followers","following_url":"https://api.github.com/users/jchodera/following{/other_user}","gists_url":"https://api.github.com/users/jchodera/gists{/gist_id}","starred_url":"https://api.github.com/users/jchodera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jchodera/subscriptions","organizations_url":"https://api.github.com/users/jchodera/orgs","repos_url":"https://api.github.com/users/jchodera/repos","events_url":"https://api.github.com/users/jchodera/events{/privacy}","received_events_url":"https://api.github.com/users/jchodera/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":29,"created_at":"2015-08-26T00:38:58Z","updated_at":"2015-08-26T19:27:39Z","closed_at":null,"body":"We've been working on debugging issues with Folding@home project 10492, a large (277K atom) system with a kinase+FAT domain running in explicit solvent with PME.  These simulations have been triggering lots of Bad State errors, which report on large discrepancies between energies or forces between `Reference` and `OpenCL` platforms.\r\n\r\nWhile we are working toward harvesting the exact configurations that cause these issues, we've used a [variant of the platform energy comparison script found in openmmtools](https://github.com/choderalab/fah-tools/blob/master/openMM_testscript.py) to compare energies among platforms on the initial `system.xml` and `state.xml` files we use for these simulations.  (I'll try to get these posted shortly)\r\n\r\nSo far, it seems that the difference between the OpenCL single-precision platform and `Reference` is  surprisingly small (~0.4 kcal/mol), but seemingly large enough to trigger Bad State errors somewhere down the road. \r\n\r\nThe majority of the error seems to be from the reciprocal-space contribution, but what is surprising is that this error does not diminish when `double` precision is used instead.  I wonder if this might be some indication of the issues that might be triggering Bad States?\r\n```\r\nRUN2 (277543 atoms)\r\n       Reference          double   -856195.726334 kcal/mol         0.000000 kcal/mol       248.287692 kcal/mol/nm         0.000000 kcal/mol/nm\r\n             CPU          single   -856196.162338 kcal/mol        -0.436004 kcal/mol       248.287627 kcal/mol/nm         0.012325 kcal/mol/nm\r\n                                 WARNING: Potential energy error (-0.436004 kcal/mol) exceeds tolerance (0.060000 kcal/mol).  Test failed.\r\n            CUDA          single   -856195.741576 kcal/mol        -0.015242 kcal/mol       248.287650 kcal/mol/nm         0.012029 kcal/mol/nm\r\n            CUDA          double   -856196.097066 kcal/mol        -0.370732 kcal/mol       248.287682 kcal/mol/nm         0.012013 kcal/mol/nm\r\n                                 WARNING: Potential energy error (-0.370732 kcal/mol) exceeds tolerance (0.060000 kcal/mol).  Test failed.\r\n          OpenCL          single   -856195.721081 kcal/mol         0.005253 kcal/mol       248.287652 kcal/mol/nm         0.012029 kcal/mol/nm\r\n          OpenCL          double   -856196.097613 kcal/mol        -0.371279 kcal/mol       248.287682 kcal/mol/nm         0.012013 kcal/mol/nm\r\n                                 WARNING: Potential energy error (-0.371279 kcal/mol) exceeds tolerance (0.060000 kcal/mol).  Test failed.\r\nBreakdown of discrepancies by Force component:\r\nHarmonicBondForce\r\n        platform       precision        potential                     error                 force mag                 rms error\r\n       Reference          double      3755.199248 kcal/mol         0.000000 kcal/mol        63.392196 kcal/mol/nm         0.000000 kcal/mol/nm\r\n             CPU          single      3755.199248 kcal/mol         0.000000 kcal/mol        63.392196 kcal/mol/nm         0.000000 kcal/mol/nm\r\n            CUDA          single      3755.200820 kcal/mol         0.001573 kcal/mol        63.392197 kcal/mol/nm         0.000166 kcal/mol/nm\r\n            CUDA          double      3755.199828 kcal/mol         0.000580 kcal/mol        63.392196 kcal/mol/nm         0.000079 kcal/mol/nm\r\n          OpenCL          single      3755.200820 kcal/mol         0.001573 kcal/mol        63.392197 kcal/mol/nm         0.000166 kcal/mol/nm\r\n          OpenCL          double      3755.199828 kcal/mol         0.000580 kcal/mol        63.392196 kcal/mol/nm         0.000079 kcal/mol/nm\r\nHarmonicAngleForce\r\n        platform       precision        potential                     error                 force mag                 rms error\r\n       Reference          double     10244.931781 kcal/mol         0.000000 kcal/mol        51.559971 kcal/mol/nm         0.000000 kcal/mol/nm\r\n             CPU          single     10244.931781 kcal/mol         0.000000 kcal/mol        51.559971 kcal/mol/nm         0.000000 kcal/mol/nm\r\n            CUDA          single     10244.933432 kcal/mol         0.001651 kcal/mol        51.559975 kcal/mol/nm         0.000051 kcal/mol/nm\r\n            CUDA          double     10244.933173 kcal/mol         0.001392 kcal/mol        51.559974 kcal/mol/nm         0.000022 kcal/mol/nm\r\n          OpenCL          single     10244.933432 kcal/mol         0.001651 kcal/mol        51.559975 kcal/mol/nm         0.000051 kcal/mol/nm\r\n          OpenCL          double     10244.933173 kcal/mol         0.001392 kcal/mol        51.559974 kcal/mol/nm         0.000022 kcal/mol/nm\r\nPeriodicTorsionForce\r\n        platform       precision        potential                     error                 force mag                 rms error\r\n       Reference          double     13825.283049 kcal/mol         0.000000 kcal/mol        17.508402 kcal/mol/nm         0.000000 kcal/mol/nm\r\n             CPU          single     13825.283049 kcal/mol         0.000000 kcal/mol        17.508402 kcal/mol/nm         0.000000 kcal/mol/nm\r\n            CUDA          single     13825.282393 kcal/mol        -0.000655 kcal/mol        17.508398 kcal/mol/nm         0.000023 kcal/mol/nm\r\n            CUDA          double     13825.283076 kcal/mol         0.000027 kcal/mol        17.508401 kcal/mol/nm         0.000005 kcal/mol/nm\r\n          OpenCL          single     13825.285339 kcal/mol         0.002291 kcal/mol        17.508400 kcal/mol/nm         0.000022 kcal/mol/nm\r\n          OpenCL          double     13825.282561 kcal/mol        -0.000487 kcal/mol        17.508399 kcal/mol/nm         0.000009 kcal/mol/nm\r\nNonbondedForce (direct)\r\n        platform       precision        potential                     error                 force mag                 rms error\r\n       Reference          double   3651288.711688 kcal/mol         0.000000 kcal/mol       235.481438 kcal/mol/nm         0.000000 kcal/mol/nm\r\n             CPU          single   3651288.624873 kcal/mol        -0.086816 kcal/mol       235.481378 kcal/mol/nm         0.000383 kcal/mol/nm\r\n            CUDA          single   3651289.121167 kcal/mol         0.409479 kcal/mol       235.481407 kcal/mol/nm         0.000444 kcal/mol/nm\r\n            CUDA          double   3651288.769640 kcal/mol         0.057951 kcal/mol       235.481441 kcal/mol/nm         0.000023 kcal/mol/nm\r\n          OpenCL          single   3651289.141455 kcal/mol         0.429766 kcal/mol       235.481409 kcal/mol/nm         0.000444 kcal/mol/nm\r\n          OpenCL          double   3651288.769640 kcal/mol         0.057951 kcal/mol       235.481441 kcal/mol/nm         0.000023 kcal/mol/nm\r\nNonbondedForce (reciprocal)\r\n        platform       precision        potential                     error                 force mag                 rms error\r\n       Reference          double  -4535309.852100 kcal/mol         0.000000 kcal/mol        10.247086 kcal/mol/nm         0.000000 kcal/mol/nm\r\n             CPU          single  -4535310.201288 kcal/mol        -0.349188 kcal/mol        10.247017 kcal/mol/nm         0.012318 kcal/mol/nm\r\n            CUDA          single  -4535310.278834 kcal/mol        -0.426734 kcal/mol        10.247016 kcal/mol/nm         0.012019 kcal/mol/nm\r\n            CUDA          double  -4535310.282783 kcal/mol        -0.430683 kcal/mol        10.247003 kcal/mol/nm         0.012013 kcal/mol/nm\r\n          OpenCL          single  -4535310.281318 kcal/mol        -0.429218 kcal/mol        10.247008 kcal/mol/nm         0.012019 kcal/mol/nm\r\n          OpenCL          double  -4535310.282816 kcal/mol        -0.430716 kcal/mol        10.247003 kcal/mol/nm         0.012013 kcal/mol/nm\r\nCMMotionRemover\r\n        platform       precision        potential                     error                 force mag                 rms error\r\n       Reference          double         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n             CPU          single         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n            CUDA          single         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n            CUDA          double         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n          OpenCL          single         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n          OpenCL          double         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\nMonteCarloBarostat\r\n        platform       precision        potential                     error                 force mag                 rms error\r\n       Reference          double         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n             CPU          single         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n            CUDA          single         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n            CUDA          double         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n          OpenCL          single         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n          OpenCL          double         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n```"},"comment":{"url":"https://api.github.com/repos/pandegroup/openmm/issues/comments/135146262","html_url":"https://github.com/pandegroup/openmm/issues/1108#issuecomment-135146262","issue_url":"https://api.github.com/repos/pandegroup/openmm/issues/1108","id":135146262,"user":{"login":"jchodera","id":3656088,"avatar_url":"https://avatars.githubusercontent.com/u/3656088?v=3","gravatar_id":"","url":"https://api.github.com/users/jchodera","html_url":"https://github.com/jchodera","followers_url":"https://api.github.com/users/jchodera/followers","following_url":"https://api.github.com/users/jchodera/following{/other_user}","gists_url":"https://api.github.com/users/jchodera/gists{/gist_id}","starred_url":"https://api.github.com/users/jchodera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jchodera/subscriptions","organizations_url":"https://api.github.com/users/jchodera/orgs","repos_url":"https://api.github.com/users/jchodera/repos","events_url":"https://api.github.com/users/jchodera/events{/privacy}","received_events_url":"https://api.github.com/users/jchodera/received_events","type":"User","site_admin":false},"created_at":"2015-08-26T19:27:39Z","updated_at":"2015-08-26T19:27:39Z","body":"Whoa. Which `openmm` are you using for this? "}}