{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/PerfKitBenchmarker/issues/717","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/PerfKitBenchmarker/issues/717/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/PerfKitBenchmarker/issues/717/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/PerfKitBenchmarker/issues/717/events","html_url":"https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/717","id":121950856,"number":717,"title":"cluster_boot seems to over-report boot time","user":{"login":"zbjornson","id":469365,"avatar_url":"https://avatars.githubusercontent.com/u/469365?v=3","gravatar_id":"","url":"https://api.github.com/users/zbjornson","html_url":"https://github.com/zbjornson","followers_url":"https://api.github.com/users/zbjornson/followers","following_url":"https://api.github.com/users/zbjornson/following{/other_user}","gists_url":"https://api.github.com/users/zbjornson/gists{/gist_id}","starred_url":"https://api.github.com/users/zbjornson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zbjornson/subscriptions","organizations_url":"https://api.github.com/users/zbjornson/orgs","repos_url":"https://api.github.com/users/zbjornson/repos","events_url":"https://api.github.com/users/zbjornson/events{/privacy}","received_events_url":"https://api.github.com/users/zbjornson/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/PerfKitBenchmarker/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/PerfKitBenchmarker/labels/P1","name":"P1","color":"fef2c0"}],"state":"open","locked":false,"assignee":{"login":"cmccoy","id":564884,"avatar_url":"https://avatars.githubusercontent.com/u/564884?v=3","gravatar_id":"","url":"https://api.github.com/users/cmccoy","html_url":"https://github.com/cmccoy","followers_url":"https://api.github.com/users/cmccoy/followers","following_url":"https://api.github.com/users/cmccoy/following{/other_user}","gists_url":"https://api.github.com/users/cmccoy/gists{/gist_id}","starred_url":"https://api.github.com/users/cmccoy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmccoy/subscriptions","organizations_url":"https://api.github.com/users/cmccoy/orgs","repos_url":"https://api.github.com/users/cmccoy/repos","events_url":"https://api.github.com/users/cmccoy/events{/privacy}","received_events_url":"https://api.github.com/users/cmccoy/received_events","type":"User","site_admin":false},"milestone":null,"comments":4,"created_at":"2015-12-13T23:02:35Z","updated_at":"2015-12-15T01:16:53Z","closed_at":null,"body":"The [code docs say that cluster_boot reports how long it takes for a VM to become responsive to SSH](https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/c1e4945ad34bd47c409ebb55d538d22f5463a643/perfkitbenchmarker/benchmarks/cluster_boot_benchmark.py#L48), but I experience much faster ssh response times for GCE than PKB is reporting.\r\n\r\nPKB for gcloud:\r\n```\r\nCLUSTER_BOOT:\r\n  num_cpus=\"1\" num_vms=\"10\" # n1-standard-1\r\n  Boot Time                            78.790919 seconds                        (machine_instance=\"0\")\r\n  Boot Time                            82.916165 seconds                        (machine_instance=\"1\")\r\n  Boot Time                            82.787568 seconds                        (machine_instance=\"2\")\r\n  Boot Time                            82.920172 seconds                        (machine_instance=\"3\")\r\n  Boot Time                            78.791941 seconds                        (machine_instance=\"4\")\r\n  ...\r\n```\r\n\r\nManually:\r\n```sh\r\n$ START=`date +%s`; CREATE=`gcloud compute instances create zb-boot-1 --format json --zone us-central1-a --image ubuntu-14-04 --boot-disk-auto-delete --boot-disk-type pd-standard --boot-disk-size 10 --machine-type n1-standard-1 --no-restart-on-failure --preemptible`; echo \"running: $((`date +%s`-$START))\"; gcloud compute --quiet ssh zb-boot-1 --command \"pwd\"; echo \"ssh response: $((`date +%s`-$START))\"; gcloud compute instances delete zb-boot-1 --quiet\r\nssh response: 52\r\n```\r\n\r\n49 to 52 seconds is pretty consistent for that image, machine type and disk type, which is to say cluster_boot over-reports by about 30 seconds. I haven't followed PKB's code too closely, but is the \"boot time\" including the time to create the network and all other resources, not just the VM? I think just the VM is the most metric useful if people are evaluating how quickly a VM can be contribute to an autoscaling group, for example. (That's my use case, at least.)\r\n\r\nFor AWS EC2, standard EBS, m4.large, ubuntu (ami-0fbce65), cluster_boot reports ~84 seconds. Manually I get 67 seconds until ping, 80 seconds until SSH. Not off by nearly as much.\r\n\r\n(Aside, cluster_boot is running stuff like `sudo apt-get update`. It doesn't seem to be timing those commands, but they waste time if you're trying to make these benchmarks run faster.)\r\n\r\n(Very happy this benchmark kit exists, thanks for releasing it!)"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/PerfKitBenchmarker/issues/comments/164610874","html_url":"https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/717#issuecomment-164610874","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/PerfKitBenchmarker/issues/717","id":164610874,"user":{"login":"cmccoy","id":564884,"avatar_url":"https://avatars.githubusercontent.com/u/564884?v=3","gravatar_id":"","url":"https://api.github.com/users/cmccoy","html_url":"https://github.com/cmccoy","followers_url":"https://api.github.com/users/cmccoy/followers","following_url":"https://api.github.com/users/cmccoy/following{/other_user}","gists_url":"https://api.github.com/users/cmccoy/gists{/gist_id}","starred_url":"https://api.github.com/users/cmccoy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmccoy/subscriptions","organizations_url":"https://api.github.com/users/cmccoy/orgs","repos_url":"https://api.github.com/users/cmccoy/repos","events_url":"https://api.github.com/users/cmccoy/events{/privacy}","received_events_url":"https://api.github.com/users/cmccoy/received_events","type":"User","site_admin":false},"created_at":"2015-12-15T01:16:53Z","updated_at":"2015-12-15T01:16:53Z","body":"Definitely. I'm working on this now."}}