{"action":"created","issue":{"url":"https://api.github.com/repos/parallella/pal/issues/203","labels_url":"https://api.github.com/repos/parallella/pal/issues/203/labels{/name}","comments_url":"https://api.github.com/repos/parallella/pal/issues/203/comments","events_url":"https://api.github.com/repos/parallella/pal/issues/203/events","html_url":"https://github.com/parallella/pal/issues/203","id":96313886,"number":203,"title":"benchmarks: No way to get reproducible results","user":{"login":"olajep","id":794072,"avatar_url":"https://avatars.githubusercontent.com/u/794072?v=3","gravatar_id":"","url":"https://api.github.com/users/olajep","html_url":"https://github.com/olajep","followers_url":"https://api.github.com/users/olajep/followers","following_url":"https://api.github.com/users/olajep/following{/other_user}","gists_url":"https://api.github.com/users/olajep/gists{/gist_id}","starred_url":"https://api.github.com/users/olajep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olajep/subscriptions","organizations_url":"https://api.github.com/users/olajep/orgs","repos_url":"https://api.github.com/users/olajep/repos","events_url":"https://api.github.com/users/olajep/events{/privacy}","received_events_url":"https://api.github.com/users/olajep/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-07-21T13:25:20Z","updated_at":"2015-07-22T19:57:26Z","closed_at":null,"body":"Variance between runs is way too high.\r\n\r\nI guess we could wrap the function in a for loop and use the lowest measurement, that would certainly improve things.\r\n\r\nBut I think we better use performance counters instead.\r\nEpiphany isn't affected by this since we already use CTIMERs there.\r\n\r\nPAPI seems to have the cross-platform support we need:\r\nhttp://icl.cs.utk.edu/papi/"},"comment":{"url":"https://api.github.com/repos/parallella/pal/issues/comments/123845468","html_url":"https://github.com/parallella/pal/issues/203#issuecomment-123845468","issue_url":"https://api.github.com/repos/parallella/pal/issues/203","id":123845468,"user":{"login":"olajep","id":794072,"avatar_url":"https://avatars.githubusercontent.com/u/794072?v=3","gravatar_id":"","url":"https://api.github.com/users/olajep","html_url":"https://github.com/olajep","followers_url":"https://api.github.com/users/olajep/followers","following_url":"https://api.github.com/users/olajep/following{/other_user}","gists_url":"https://api.github.com/users/olajep/gists{/gist_id}","starred_url":"https://api.github.com/users/olajep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olajep/subscriptions","organizations_url":"https://api.github.com/users/olajep/orgs","repos_url":"https://api.github.com/users/olajep/repos","events_url":"https://api.github.com/users/olajep/events{/privacy}","received_events_url":"https://api.github.com/users/olajep/received_events","type":"User","site_admin":false},"created_at":"2015-07-22T19:57:26Z","updated_at":"2015-07-22T19:57:26Z","body":"@lchamon\n\nOn 2015-07-22 02:51, Chamon wrote:\n>\n> @olajep <https://github.com/olajep> What do you think about timing several runs of the functions? Most of them run in the ns, maybe running them 1000 times (for example) would improve the precision of the measurements.\n>\nYes, that will certainly improve things. I did some testing now and even with 32000 iterations there can be a 20 percent diff between two runs (most are pretty close however). Which is a lot better than before (>100% !). And 32000 iterations takes way too long for several functions. Benchmarking one of the image functions takes almost two minutes.\n>\n> Also, running the benchmark a few times is a good idea to get some statistics. Maybe report minimum, median and maximum run times.\n>\nI'm not so sure about that, we should only care about the lowest measurement. Everything else is noise (e.g., context switches, some other process evicting our data from L2 cache ...).\n>\n> What I mean is something like:\n>\n> for (i = 0; i < 100; i++) {\n>     item_preface(&data[i], ...);\n>     for (j = 0; j < 1000; j++) {\n>         fun()\n>     }\n>     item_done(&data[i], ...);\n> }\nYup, we need the loop, no question about that. The question is how many iterations.\n>\n> PAPI looks cool... Maybe we could take only the ARM/x86 parts (I see they don't support Windows anymore, not sure if that would be an issue).\n>\nI believe that benchmarking using clock cycles instead will give more stable results.\nWe also need to size the benchmarks so they:\ni) fit in cache and (we don't want to benchmark the memory subsystem) and\nii) one function call should be short enough to not always be preempted by the kernel.\n\nI think that should do it.\n\nCheers,\nOla\n"}}