{"action":"created","comment":{"url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/comments/44645802","id":44645802,"diff_hunk":"@@ -0,0 +1,190 @@\n+from datetime import datetime\n+import json\n+from os import environ, getcwd\n+from platform import node, platform\n+from sys import stdout\n+\n+from eliot import start_action\n+from eliot.twisted import DeferredContext\n+\n+from twisted.python.components import proxyForInterface\n+from twisted.python.filepath import FilePath\n+from twisted.internet.task import cooperate\n+from twisted.internet.defer import maybeDeferred, Deferred\n+\n+from flocker import __version__ as flocker_client_version\n+from flocker.apiclient import (\n+    IFlockerAPIV1Client, FakeFlockerClient, FlockerClient,\n+)\n+from flocker.common import gather_deferreds\n+\n+from benchmark_operations import get_operation\n+from benchmark_measurements import get_measurement\n+from benchmark_scenarios import get_scenario\n+\n+\n+def benchmark(measure, operation, scenario, n=3):\n+    \"\"\"\n+    Perform sampling of the operation.\n+\n+    :param measure: A quantity to measure.\n+    :param operation: An operation to perform.","path":"benchmark/benchmark_driver.py","position":31,"original_position":31,"commit_id":"4cb5b53fe4e2eb44e2fa09af7f387d6201cb3904","original_commit_id":"4cb5b53fe4e2eb44e2fa09af7f387d6201cb3904","user":{"login":"jml","id":125674,"avatar_url":"https://avatars.githubusercontent.com/u/125674?v=3","gravatar_id":"","url":"https://api.github.com/users/jml","html_url":"https://github.com/jml","followers_url":"https://api.github.com/users/jml/followers","following_url":"https://api.github.com/users/jml/following{/other_user}","gists_url":"https://api.github.com/users/jml/gists{/gist_id}","starred_url":"https://api.github.com/users/jml/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jml/subscriptions","organizations_url":"https://api.github.com/users/jml/orgs","repos_url":"https://api.github.com/users/jml/repos","events_url":"https://api.github.com/users/jml/events{/privacy}","received_events_url":"https://api.github.com/users/jml/received_events","type":"User","site_admin":false},"body":"I do understand better, the operation docs are very helpful in this regard. Please update the docstring to say that operation is an `IOperation`.\r\n\r\nPlease also consider renaming `n` to `num_samples`.","created_at":"2015-11-12T11:15:38Z","updated_at":"2015-11-12T11:15:38Z","html_url":"https://github.com/ClusterHQ/flocker/pull/2162#discussion_r44645802","pull_request_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2162","_links":{"self":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/comments/44645802"},"html":{"href":"https://github.com/ClusterHQ/flocker/pull/2162#discussion_r44645802"},"pull_request":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2162"}}},"pull_request":{"url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2162","id":50416503,"html_url":"https://github.com/ClusterHQ/flocker/pull/2162","diff_url":"https://github.com/ClusterHQ/flocker/pull/2162.diff","patch_url":"https://github.com/ClusterHQ/flocker/pull/2162.patch","issue_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2162","number":2162,"state":"open","locked":false,"title":"[FLOC-3288] Benchmark driver","user":{"login":"jongiddy","id":3735327,"avatar_url":"https://avatars.githubusercontent.com/u/3735327?v=3","gravatar_id":"","url":"https://api.github.com/users/jongiddy","html_url":"https://github.com/jongiddy","followers_url":"https://api.github.com/users/jongiddy/followers","following_url":"https://api.github.com/users/jongiddy/following{/other_user}","gists_url":"https://api.github.com/users/jongiddy/gists{/gist_id}","starred_url":"https://api.github.com/users/jongiddy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jongiddy/subscriptions","organizations_url":"https://api.github.com/users/jongiddy/orgs","repos_url":"https://api.github.com/users/jongiddy/repos","events_url":"https://api.github.com/users/jongiddy/events{/privacy}","received_events_url":"https://api.github.com/users/jongiddy/received_events","type":"User","site_admin":false},"body":"This PR creates a basic benchmarking command. Using an existing Flocker cluster, the command allows the user to combine a scenario (e.g. specified rate of external load on the control server), an operation to be measured (read request, write request), and a form of measurement (wallclock time, CPU time, log file size).\r\n\r\nThe driver will setup the load scenario, and then run the operation, taking measurements before and after the operation runs.  The measurement is currently sampled 3 times.  The sample results are reported to stdout as a JSON structure.\r\n\r\nThe current PR provides the basic framework and one or two examples of each parameter. The current scenario is \"no load\", the current operations are \"no-op\" and \"read-request\", and the current measurement is \"wallclock\" time.\r\n\r\nAdditional issues in FLOC-2748 will introduce more values for these parameters. Untested versions of these additional values can be seen in the spike branch https://github.com/ClusterHQ/flocker/tree/benchmark-prototype-with-scenarios.\r\n\r\nWith no Flocker cluster, run the benchmark using:\r\n```\r\npython -m benchmark.benchmark_control --operation nop\r\n```\r\n\r\nThis will display a JSON stanza like:\r\n```json\r\n{\"client\": {\"username\": \"jon\", \"flocker_version\": \"1.6.1+111.g4cb5b53\", \"working_directory\": \"/home/jon/projects/clusterhq/flocker\", \"nodename\": \"ecuador\", \"platform\": \"Linux-3.13.0-66-generic-x86_64-with-Ubuntu-14.04-trusty\", \"date\": \"2015-11-11T18:11:19.173954\"}, \"measurement\": \"wallclock\", \"metric_name\": \"nop\", \"samples\": [{\"value\": 1.0967254638671875e-05, \"success\": true}, {\"value\": 6.9141387939453125e-06, \"success\": true}, {\"value\": 7.152557373046875e-06, \"success\": true}], \"server\": {\"flocker_version\": \"1.6.1+111.g4cb5b53\"}}\r\n```\r\n\r\nNote, as well as the final JSON output to stdout, the script displays logging information to stderr.\r\n\r\nIf you have a Flocker cluster handy (and who doesn't), to get a genuine benchmark result, run:\r\n```\r\npython -m benchmark.benchmark_control --control=${FLOCKER_ACCEPTANCE_CONTROL_NODE} --certs=${FLOCKER_ACCEPTANCE_API_CERTIFICATES_PATH}\r\n```\r\nto get a genuine benchmark result.\r\n","created_at":"2015-11-11T18:15:04Z","updated_at":"2015-11-12T11:15:38Z","closed_at":null,"merged_at":null,"merge_commit_sha":"5489a5babaf250a1681a5b7b50608ea18beeb7dc","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2162/commits","review_comments_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2162/comments","review_comment_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/comments{/number}","comments_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2162/comments","statuses_url":"https://api.github.com/repos/ClusterHQ/flocker/statuses/4cb5b53fe4e2eb44e2fa09af7f387d6201cb3904","head":{"label":"ClusterHQ:benchmark-driver-2-FLOC-3288","ref":"benchmark-driver-2-FLOC-3288","sha":"4cb5b53fe4e2eb44e2fa09af7f387d6201cb3904","user":{"login":"ClusterHQ","id":7913449,"avatar_url":"https://avatars.githubusercontent.com/u/7913449?v=3","gravatar_id":"","url":"https://api.github.com/users/ClusterHQ","html_url":"https://github.com/ClusterHQ","followers_url":"https://api.github.com/users/ClusterHQ/followers","following_url":"https://api.github.com/users/ClusterHQ/following{/other_user}","gists_url":"https://api.github.com/users/ClusterHQ/gists{/gist_id}","starred_url":"https://api.github.com/users/ClusterHQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClusterHQ/subscriptions","organizations_url":"https://api.github.com/users/ClusterHQ/orgs","repos_url":"https://api.github.com/users/ClusterHQ/repos","events_url":"https://api.github.com/users/ClusterHQ/events{/privacy}","received_events_url":"https://api.github.com/users/ClusterHQ/received_events","type":"Organization","site_admin":false},"repo":{"id":19248542,"name":"flocker","full_name":"ClusterHQ/flocker","owner":{"login":"ClusterHQ","id":7913449,"avatar_url":"https://avatars.githubusercontent.com/u/7913449?v=3","gravatar_id":"","url":"https://api.github.com/users/ClusterHQ","html_url":"https://github.com/ClusterHQ","followers_url":"https://api.github.com/users/ClusterHQ/followers","following_url":"https://api.github.com/users/ClusterHQ/following{/other_user}","gists_url":"https://api.github.com/users/ClusterHQ/gists{/gist_id}","starred_url":"https://api.github.com/users/ClusterHQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClusterHQ/subscriptions","organizations_url":"https://api.github.com/users/ClusterHQ/orgs","repos_url":"https://api.github.com/users/ClusterHQ/repos","events_url":"https://api.github.com/users/ClusterHQ/events{/privacy}","received_events_url":"https://api.github.com/users/ClusterHQ/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ClusterHQ/flocker","description":"Container data volume manager for your Dockerized application","fork":false,"url":"https://api.github.com/repos/ClusterHQ/flocker","forks_url":"https://api.github.com/repos/ClusterHQ/flocker/forks","keys_url":"https://api.github.com/repos/ClusterHQ/flocker/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ClusterHQ/flocker/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ClusterHQ/flocker/teams","hooks_url":"https://api.github.com/repos/ClusterHQ/flocker/hooks","issue_events_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/events{/number}","events_url":"https://api.github.com/repos/ClusterHQ/flocker/events","assignees_url":"https://api.github.com/repos/ClusterHQ/flocker/assignees{/user}","branches_url":"https://api.github.com/repos/ClusterHQ/flocker/branches{/branch}","tags_url":"https://api.github.com/repos/ClusterHQ/flocker/tags","blobs_url":"https://api.github.com/repos/ClusterHQ/flocker/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ClusterHQ/flocker/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ClusterHQ/flocker/git/refs{/sha}","trees_url":"https://api.github.com/repos/ClusterHQ/flocker/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ClusterHQ/flocker/statuses/{sha}","languages_url":"https://api.github.com/repos/ClusterHQ/flocker/languages","stargazers_url":"https://api.github.com/repos/ClusterHQ/flocker/stargazers","contributors_url":"https://api.github.com/repos/ClusterHQ/flocker/contributors","subscribers_url":"https://api.github.com/repos/ClusterHQ/flocker/subscribers","subscription_url":"https://api.github.com/repos/ClusterHQ/flocker/subscription","commits_url":"https://api.github.com/repos/ClusterHQ/flocker/commits{/sha}","git_commits_url":"https://api.github.com/repos/ClusterHQ/flocker/git/commits{/sha}","comments_url":"https://api.github.com/repos/ClusterHQ/flocker/comments{/number}","issue_comment_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/comments{/number}","contents_url":"https://api.github.com/repos/ClusterHQ/flocker/contents/{+path}","compare_url":"https://api.github.com/repos/ClusterHQ/flocker/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ClusterHQ/flocker/merges","archive_url":"https://api.github.com/repos/ClusterHQ/flocker/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ClusterHQ/flocker/downloads","issues_url":"https://api.github.com/repos/ClusterHQ/flocker/issues{/number}","pulls_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls{/number}","milestones_url":"https://api.github.com/repos/ClusterHQ/flocker/milestones{/number}","notifications_url":"https://api.github.com/repos/ClusterHQ/flocker/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ClusterHQ/flocker/labels{/name}","releases_url":"https://api.github.com/repos/ClusterHQ/flocker/releases{/id}","created_at":"2014-04-28T18:02:14Z","updated_at":"2015-11-11T17:41:02Z","pushed_at":"2015-11-12T10:37:33Z","git_url":"git://github.com/ClusterHQ/flocker.git","ssh_url":"git@github.com:ClusterHQ/flocker.git","clone_url":"https://github.com/ClusterHQ/flocker.git","svn_url":"https://github.com/ClusterHQ/flocker","homepage":"https://clusterhq.com","size":71220,"stargazers_count":1945,"watchers_count":1945,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":142,"mirror_url":null,"open_issues_count":31,"forks":142,"open_issues":31,"watchers":1945,"default_branch":"master"}},"base":{"label":"ClusterHQ:master","ref":"master","sha":"83de9ee0884140d4f66a603e1cc0b8f2962d05c7","user":{"login":"ClusterHQ","id":7913449,"avatar_url":"https://avatars.githubusercontent.com/u/7913449?v=3","gravatar_id":"","url":"https://api.github.com/users/ClusterHQ","html_url":"https://github.com/ClusterHQ","followers_url":"https://api.github.com/users/ClusterHQ/followers","following_url":"https://api.github.com/users/ClusterHQ/following{/other_user}","gists_url":"https://api.github.com/users/ClusterHQ/gists{/gist_id}","starred_url":"https://api.github.com/users/ClusterHQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClusterHQ/subscriptions","organizations_url":"https://api.github.com/users/ClusterHQ/orgs","repos_url":"https://api.github.com/users/ClusterHQ/repos","events_url":"https://api.github.com/users/ClusterHQ/events{/privacy}","received_events_url":"https://api.github.com/users/ClusterHQ/received_events","type":"Organization","site_admin":false},"repo":{"id":19248542,"name":"flocker","full_name":"ClusterHQ/flocker","owner":{"login":"ClusterHQ","id":7913449,"avatar_url":"https://avatars.githubusercontent.com/u/7913449?v=3","gravatar_id":"","url":"https://api.github.com/users/ClusterHQ","html_url":"https://github.com/ClusterHQ","followers_url":"https://api.github.com/users/ClusterHQ/followers","following_url":"https://api.github.com/users/ClusterHQ/following{/other_user}","gists_url":"https://api.github.com/users/ClusterHQ/gists{/gist_id}","starred_url":"https://api.github.com/users/ClusterHQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClusterHQ/subscriptions","organizations_url":"https://api.github.com/users/ClusterHQ/orgs","repos_url":"https://api.github.com/users/ClusterHQ/repos","events_url":"https://api.github.com/users/ClusterHQ/events{/privacy}","received_events_url":"https://api.github.com/users/ClusterHQ/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ClusterHQ/flocker","description":"Container data volume manager for your Dockerized application","fork":false,"url":"https://api.github.com/repos/ClusterHQ/flocker","forks_url":"https://api.github.com/repos/ClusterHQ/flocker/forks","keys_url":"https://api.github.com/repos/ClusterHQ/flocker/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ClusterHQ/flocker/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ClusterHQ/flocker/teams","hooks_url":"https://api.github.com/repos/ClusterHQ/flocker/hooks","issue_events_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/events{/number}","events_url":"https://api.github.com/repos/ClusterHQ/flocker/events","assignees_url":"https://api.github.com/repos/ClusterHQ/flocker/assignees{/user}","branches_url":"https://api.github.com/repos/ClusterHQ/flocker/branches{/branch}","tags_url":"https://api.github.com/repos/ClusterHQ/flocker/tags","blobs_url":"https://api.github.com/repos/ClusterHQ/flocker/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ClusterHQ/flocker/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ClusterHQ/flocker/git/refs{/sha}","trees_url":"https://api.github.com/repos/ClusterHQ/flocker/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ClusterHQ/flocker/statuses/{sha}","languages_url":"https://api.github.com/repos/ClusterHQ/flocker/languages","stargazers_url":"https://api.github.com/repos/ClusterHQ/flocker/stargazers","contributors_url":"https://api.github.com/repos/ClusterHQ/flocker/contributors","subscribers_url":"https://api.github.com/repos/ClusterHQ/flocker/subscribers","subscription_url":"https://api.github.com/repos/ClusterHQ/flocker/subscription","commits_url":"https://api.github.com/repos/ClusterHQ/flocker/commits{/sha}","git_commits_url":"https://api.github.com/repos/ClusterHQ/flocker/git/commits{/sha}","comments_url":"https://api.github.com/repos/ClusterHQ/flocker/comments{/number}","issue_comment_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/comments{/number}","contents_url":"https://api.github.com/repos/ClusterHQ/flocker/contents/{+path}","compare_url":"https://api.github.com/repos/ClusterHQ/flocker/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ClusterHQ/flocker/merges","archive_url":"https://api.github.com/repos/ClusterHQ/flocker/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ClusterHQ/flocker/downloads","issues_url":"https://api.github.com/repos/ClusterHQ/flocker/issues{/number}","pulls_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls{/number}","milestones_url":"https://api.github.com/repos/ClusterHQ/flocker/milestones{/number}","notifications_url":"https://api.github.com/repos/ClusterHQ/flocker/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ClusterHQ/flocker/labels{/name}","releases_url":"https://api.github.com/repos/ClusterHQ/flocker/releases{/id}","created_at":"2014-04-28T18:02:14Z","updated_at":"2015-11-11T17:41:02Z","pushed_at":"2015-11-12T10:37:33Z","git_url":"git://github.com/ClusterHQ/flocker.git","ssh_url":"git@github.com:ClusterHQ/flocker.git","clone_url":"https://github.com/ClusterHQ/flocker.git","svn_url":"https://github.com/ClusterHQ/flocker","homepage":"https://clusterhq.com","size":71220,"stargazers_count":1945,"watchers_count":1945,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":142,"mirror_url":null,"open_issues_count":31,"forks":142,"open_issues":31,"watchers":1945,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2162"},"html":{"href":"https://github.com/ClusterHQ/flocker/pull/2162"},"issue":{"href":"https://api.github.com/repos/ClusterHQ/flocker/issues/2162"},"comments":{"href":"https://api.github.com/repos/ClusterHQ/flocker/issues/2162/comments"},"review_comments":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2162/comments"},"review_comment":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2162/commits"},"statuses":{"href":"https://api.github.com/repos/ClusterHQ/flocker/statuses/4cb5b53fe4e2eb44e2fa09af7f387d6201cb3904"}}}}