{"action":"created","issue":{"url":"https://api.github.com/repos/pandegroup/openmm/issues/1012","labels_url":"https://api.github.com/repos/pandegroup/openmm/issues/1012/labels{/name}","comments_url":"https://api.github.com/repos/pandegroup/openmm/issues/1012/comments","events_url":"https://api.github.com/repos/pandegroup/openmm/issues/1012/events","html_url":"https://github.com/pandegroup/openmm/issues/1012","id":93163499,"number":1012,"title":"Significant potential energy discrepancy on CUDA and OpenCL platforms for DHFR single-precision","user":{"login":"jchodera","id":3656088,"avatar_url":"https://avatars.githubusercontent.com/u/3656088?v=3","gravatar_id":"","url":"https://api.github.com/users/jchodera","html_url":"https://github.com/jchodera","followers_url":"https://api.github.com/users/jchodera/followers","following_url":"https://api.github.com/users/jchodera/following{/other_user}","gists_url":"https://api.github.com/users/jchodera/gists{/gist_id}","starred_url":"https://api.github.com/users/jchodera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jchodera/subscriptions","organizations_url":"https://api.github.com/users/jchodera/orgs","repos_url":"https://api.github.com/users/jchodera/repos","events_url":"https://api.github.com/users/jchodera/events{/privacy}","received_events_url":"https://api.github.com/users/jchodera/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-07-06T01:00:29Z","updated_at":"2015-07-29T22:15:00Z","closed_at":"2015-07-29T20:29:35Z","body":"The [`test-openmm-platforms` comparison I ran](https://gist.github.com/jchodera/a55116ca82dd5750af53#file-test-openmm-platforms-6-3-0-dev-38076a5-gtx680-log-L250-L259) shows significant discrepancies for the DHFR explicit system for single-precision CUDA and OpenCL, originating from the `NonbondedForce` direct interaction, suggesting it might be worth revisiting the `erf` approximations used for CUDA and OpenCL (recall #907):\r\n```\r\n2015-07-05 19:37:03,264: DHFRExplicit\r\n2015-07-05 19:37:20,402:        Reference          double    -72297.598527 kcal/mol         0.000000 kcal/mol       246.160114 kcal/mol/nm         0.000000 kcal/mol/nm\r\n2015-07-05 19:37:21,244:              CPU          single    -72297.628317 kcal/mol        -0.029789 kcal/mol       246.159976 kcal/mol/nm         0.009107 kcal/mol/nm\r\n2015-07-05 19:37:31,427:             CUDA          single    -72297.480891 kcal/mol         0.117637 kcal/mol       246.159649 kcal/mol/nm         0.023276 kcal/mol/nm\r\n2015-07-05 19:37:31,428:                                  WARNING: Potential energy error (0.117637 kcal/mol) exceeds tolerance (0.060000 kcal/mol).  Test failed.\r\n2015-07-05 19:37:42,836:             CUDA          double    -72297.594358 kcal/mol         0.004169 kcal/mol       246.160270 kcal/mol/nm         0.000496 kcal/mol/nm\r\n2015-07-05 19:37:46,368:           OpenCL          single    -72297.487854 kcal/mol         0.110673 kcal/mol       246.159649 kcal/mol/nm         0.023275 kcal/mol/nm\r\n2015-07-05 19:37:46,368:                                  WARNING: Potential energy error (0.110673 kcal/mol) exceeds tolerance (0.060000 kcal/mol).  Test failed.\r\n2015-07-05 19:37:51,072:           OpenCL          double    -72297.594509 kcal/mol         0.004018 kcal/mol       246.160269 kcal/mol/nm         0.000503 kcal/mol/nm\r\n2015-07-05 19:37:51,072: Writing failed test system to 'DHFRExplicit'.{system,state}.xml ...\r\n2015-07-05 19:37:53,483: Breakdown of discrepancies by Force component:\r\n2015-07-05 19:37:53,483: HarmonicBondForce\r\n2015-07-05 19:37:55,148:         platform       precision        potential                     error                 force mag                 rms error\r\n2015-07-05 19:37:56,643:        Reference          double       397.289733 kcal/mol         0.000000 kcal/mol        68.788409 kcal/mol/nm         0.000000 kcal/mol/nm\r\n2015-07-05 19:37:57,273:              CPU          single       397.289733 kcal/mol         0.000000 kcal/mol        68.788409 kcal/mol/nm         0.000000 kcal/mol/nm\r\n2015-07-05 19:37:58,636:             CUDA          single       397.289054 kcal/mol        -0.000678 kcal/mol        68.788283 kcal/mol/nm         0.022908 kcal/mol/nm\r\n2015-07-05 19:38:00,138:             CUDA          double       397.289828 kcal/mol         0.000095 kcal/mol        68.788407 kcal/mol/nm         0.000497 kcal/mol/nm\r\n2015-07-05 19:38:01,126:           OpenCL          single       397.289054 kcal/mol        -0.000678 kcal/mol        68.788283 kcal/mol/nm         0.022908 kcal/mol/nm\r\n2015-07-05 19:38:02,153:           OpenCL          double       397.289828 kcal/mol         0.000095 kcal/mol        68.788407 kcal/mol/nm         0.000497 kcal/mol/nm\r\n2015-07-05 19:38:02,153: HarmonicAngleForce\r\n2015-07-05 19:38:03,643:         platform       precision        potential                     error                 force mag                 rms error\r\n2015-07-05 19:38:05,152:        Reference          double      1183.687267 kcal/mol         0.000000 kcal/mol        59.679170 kcal/mol/nm         0.000000 kcal/mol/nm\r\n2015-07-05 19:38:05,777:              CPU          single      1183.687267 kcal/mol         0.000000 kcal/mol        59.679170 kcal/mol/nm         0.000000 kcal/mol/nm\r\n2015-07-05 19:38:06,550:             CUDA          single      1183.687471 kcal/mol         0.000204 kcal/mol        59.679188 kcal/mol/nm         0.001586 kcal/mol/nm\r\n2015-07-05 19:38:07,128:             CUDA          double      1183.687418 kcal/mol         0.000150 kcal/mol        59.679173 kcal/mol/nm         0.000021 kcal/mol/nm\r\n2015-07-05 19:38:07,697:           OpenCL          single      1183.687471 kcal/mol         0.000204 kcal/mol        59.679188 kcal/mol/nm         0.001586 kcal/mol/nm\r\n2015-07-05 19:38:08,276:           OpenCL          double      1183.687418 kcal/mol         0.000150 kcal/mol        59.679173 kcal/mol/nm         0.000021 kcal/mol/nm\r\n2015-07-05 19:38:08,276: PeriodicTorsionForce\r\n2015-07-05 19:38:09,962:         platform       precision        potential                     error                 force mag                 rms error\r\n2015-07-05 19:38:11,461:        Reference          double       965.215433 kcal/mol         0.000000 kcal/mol        19.387942 kcal/mol/nm         0.000000 kcal/mol/nm\r\n2015-07-05 19:38:12,082:              CPU          single       965.215433 kcal/mol         0.000000 kcal/mol        19.387942 kcal/mol/nm         0.000000 kcal/mol/nm\r\n2015-07-05 19:38:12,650:             CUDA          single       965.215074 kcal/mol        -0.000359 kcal/mol        19.387922 kcal/mol/nm         0.000483 kcal/mol/nm\r\n2015-07-05 19:38:13,229:             CUDA          double       965.215413 kcal/mol        -0.000019 kcal/mol        19.387942 kcal/mol/nm         0.000004 kcal/mol/nm\r\n2015-07-05 19:38:14,001:           OpenCL          single       965.215466 kcal/mol         0.000033 kcal/mol        19.387922 kcal/mol/nm         0.000484 kcal/mol/nm\r\n2015-07-05 19:38:14,576:           OpenCL          double       965.215339 kcal/mol        -0.000094 kcal/mol        19.387938 kcal/mol/nm         0.000011 kcal/mol/nm\r\n2015-07-05 19:38:14,576: NonbondedForce (direct)\r\n2015-07-05 19:38:16,818:         platform       precision        potential                     error                 force mag                 rms error\r\n2015-07-05 19:38:19,048:        Reference          double    393427.611382 kcal/mol         0.000000 kcal/mol       230.748609 kcal/mol/nm         0.000000 kcal/mol/nm\r\n2015-07-05 19:38:19,921:              CPU          single    393427.592666 kcal/mol        -0.018716 kcal/mol       230.748537 kcal/mol/nm         0.001099 kcal/mol/nm\r\n2015-07-05 19:38:20,508:             CUDA          single    393427.724965 kcal/mol         0.113583 kcal/mol       230.748588 kcal/mol/nm         0.001135 kcal/mol/nm\r\n2015-07-05 19:38:21,111:             CUDA          double    393427.615331 kcal/mol         0.003949 kcal/mol       230.748612 kcal/mol/nm         0.000024 kcal/mol/nm\r\n2015-07-05 19:38:21,691:           OpenCL          single    393427.721981 kcal/mol         0.110599 kcal/mol       230.748590 kcal/mol/nm         0.001134 kcal/mol/nm\r\n2015-07-05 19:38:22,305:           OpenCL          double    393427.615240 kcal/mol         0.003858 kcal/mol       230.748612 kcal/mol/nm         0.000087 kcal/mol/nm\r\n2015-07-05 19:38:22,305: NonbondedForce (reciprocal)\r\n2015-07-05 19:38:24,122:         platform       precision        potential                     error                 force mag                 rms error\r\n2015-07-05 19:38:25,722:        Reference          double   -468271.402342 kcal/mol         0.000000 kcal/mol        14.959687 kcal/mol/nm         0.000000 kcal/mol/nm\r\n2015-07-05 19:38:26,383:              CPU          single   -468271.413414 kcal/mol        -0.011073 kcal/mol        14.959573 kcal/mol/nm         0.009028 kcal/mol/nm\r\n2015-07-05 19:38:26,956:             CUDA          single   -468271.401981 kcal/mol         0.000361 kcal/mol        14.959691 kcal/mol/nm         0.000431 kcal/mol/nm\r\n2015-07-05 19:38:27,528:             CUDA          double   -468271.402348 kcal/mol        -0.000006 kcal/mol        14.959687 kcal/mol/nm         0.000005 kcal/mol/nm\r\n2015-07-05 19:38:28,284:           OpenCL          single   -468271.402484 kcal/mol        -0.000142 kcal/mol        14.959681 kcal/mol/nm         0.000429 kcal/mol/nm\r\n2015-07-05 19:38:28,861:           OpenCL          double   -468271.402334 kcal/mol         0.000008 kcal/mol        14.959687 kcal/mol/nm         0.000006 kcal/mol/nm\r\n2015-07-05 19:38:28,861: CMMotionRemover\r\n2015-07-05 19:38:30,360:         platform       precision        potential                     error                 force mag                 rms error\r\n2015-07-05 19:38:31,871:        Reference          double         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n2015-07-05 19:38:32,687:              CPU          single         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n2015-07-05 19:38:33,267:             CUDA          single         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n2015-07-05 19:38:33,849:             CUDA          double         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n2015-07-05 19:38:34,415:           OpenCL          single         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n2015-07-05 19:38:35,186:           OpenCL          double         0.000000 kcal/mol         0.000000 kcal/mol         0.000000 kcal/mol/nm         0.000000 kcal/mol/nm\r\n```"},"comment":{"url":"https://api.github.com/repos/pandegroup/openmm/issues/comments/126112200","html_url":"https://github.com/pandegroup/openmm/issues/1012#issuecomment-126112200","issue_url":"https://api.github.com/repos/pandegroup/openmm/issues/1012","id":126112200,"user":{"login":"swails","id":779022,"avatar_url":"https://avatars.githubusercontent.com/u/779022?v=3","gravatar_id":"","url":"https://api.github.com/users/swails","html_url":"https://github.com/swails","followers_url":"https://api.github.com/users/swails/followers","following_url":"https://api.github.com/users/swails/following{/other_user}","gists_url":"https://api.github.com/users/swails/gists{/gist_id}","starred_url":"https://api.github.com/users/swails/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/swails/subscriptions","organizations_url":"https://api.github.com/users/swails/orgs","repos_url":"https://api.github.com/users/swails/repos","events_url":"https://api.github.com/users/swails/events{/privacy}","received_events_url":"https://api.github.com/users/swails/received_events","type":"User","site_admin":false},"created_at":"2015-07-29T22:15:00Z","updated_at":"2015-07-29T22:15:00Z","body":"> And yes, double precision works fine, but is much slower.\r\n\r\nWell we seem to be operating near the limit of single precision capabilities, so it wouldn't surprise me if double precision was a necessary cost for reweighting-level accuracy/precision.  I'd obviously love to be proven wrong, though (maybe via fixed precision accumulators?)"}}