{"action":"opened","issue":{"url":"https://api.github.com/repos/pothosware/pothos/issues/5","labels_url":"https://api.github.com/repos/pothosware/pothos/issues/5/labels{/name}","comments_url":"https://api.github.com/repos/pothosware/pothos/issues/5/comments","events_url":"https://api.github.com/repos/pothosware/pothos/issues/5/events","html_url":"https://github.com/pothosware/pothos/issues/5","id":54735976,"number":5,"title":"Launch pothos-benchmark toolkit","user":{"login":"guruofquality","id":184117,"avatar_url":"https://avatars.githubusercontent.com/u/184117?v=3","gravatar_id":"","url":"https://api.github.com/users/guruofquality","html_url":"https://github.com/guruofquality","followers_url":"https://api.github.com/users/guruofquality/followers","following_url":"https://api.github.com/users/guruofquality/following{/other_user}","gists_url":"https://api.github.com/users/guruofquality/gists{/gist_id}","starred_url":"https://api.github.com/users/guruofquality/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guruofquality/subscriptions","organizations_url":"https://api.github.com/users/guruofquality/orgs","repos_url":"https://api.github.com/users/guruofquality/repos","events_url":"https://api.github.com/users/guruofquality/events{/privacy}","received_events_url":"https://api.github.com/users/guruofquality/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pothosware/pothos/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":{"login":"guruofquality","id":184117,"avatar_url":"https://avatars.githubusercontent.com/u/184117?v=3","gravatar_id":"","url":"https://api.github.com/users/guruofquality","html_url":"https://github.com/guruofquality","followers_url":"https://api.github.com/users/guruofquality/followers","following_url":"https://api.github.com/users/guruofquality/following{/other_user}","gists_url":"https://api.github.com/users/guruofquality/gists{/gist_id}","starred_url":"https://api.github.com/users/guruofquality/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guruofquality/subscriptions","organizations_url":"https://api.github.com/users/guruofquality/orgs","repos_url":"https://api.github.com/users/guruofquality/repos","events_url":"https://api.github.com/users/guruofquality/events{/privacy}","received_events_url":"https://api.github.com/users/guruofquality/received_events","type":"User","site_admin":false},"milestone":null,"comments":0,"created_at":"2015-01-19T08:06:10Z","updated_at":"2015-01-19T08:06:10Z","closed_at":null,"body":"With the new feature of JSON topologies and JSON stats reporting, and nicely bundled up into the PothoUtil executable: https://github.com/pothosware/pothos-library/wiki/PothosUtilGuide#running-a-json-topology ... it should be possible to make a python-based benchmark suite that executes a variety of benchmarks, collecting the data, and plotting it in matplotlib to create nice comparison barcharts.\r\n\r\nThis helps to establish a very encompasing baseline to determine how future features and performance work affects *anything and everything*. Most of the benchmarks will probably use passthrough blocks like router, gateway, black hole, infinite source... so that scheduler overhead is the dominating factor in the benchmarks.\r\n\r\nInteresting things to vary:\r\n* machines, arm, x86, NUMA nodes\r\n* thread pool vs thread per actor\r\n* waiting on CVs vs spining\r\n* extreme topologies - linear-chain, massive fan-in, fan-out, hyper-mesh\r\n\r\nClose this issue when the benchmark toolkit is created and in a mostly useful state."}}