{"action":"created","comment":{"url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/comments/45812621","id":45812621,"diff_hunk":"@@ -486,10 +486,10 @@ branches.each {\n     wrappers {\n       timestamps()\n       colorizeOutput()\n-      {# lock down the multijob to 45 minutes so that 'stuck' jobs won't\n+      {# lock down the multijob to 60 minutes so that 'stuck' jobs won't\n           block future runs                                                  #}\n       timeout {\n-          absolute(45)\n+          absolute(60)\n           }","path":"jobs.groovy.j2","position":10,"original_position":10,"commit_id":"feb3e3d9ca966308ae4af4ccfc6aaf070d7c7677","original_commit_id":"feb3e3d9ca966308ae4af4ccfc6aaf070d7c7677","user":{"login":"Azulinho","id":851757,"avatar_url":"https://avatars.githubusercontent.com/u/851757?v=3","gravatar_id":"","url":"https://api.github.com/users/Azulinho","html_url":"https://github.com/Azulinho","followers_url":"https://api.github.com/users/Azulinho/followers","following_url":"https://api.github.com/users/Azulinho/following{/other_user}","gists_url":"https://api.github.com/users/Azulinho/gists{/gist_id}","starred_url":"https://api.github.com/users/Azulinho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Azulinho/subscriptions","organizations_url":"https://api.github.com/users/Azulinho/orgs","repos_url":"https://api.github.com/users/Azulinho/repos","events_url":"https://api.github.com/users/Azulinho/events{/privacy}","received_events_url":"https://api.github.com/users/Azulinho/received_events","type":"User","site_admin":false},"body":"the multijob is the main job that executes all the other jobs, it has a larger timeout as it needs to wait for slaves to become available, and to copy artifacts back from the child jobs.","created_at":"2015-11-24T23:51:54Z","updated_at":"2015-11-24T23:51:54Z","html_url":"https://github.com/ClusterHQ/flocker/pull/2229#discussion_r45812621","pull_request_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2229","_links":{"self":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/comments/45812621"},"html":{"href":"https://github.com/ClusterHQ/flocker/pull/2229#discussion_r45812621"},"pull_request":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2229"}}},"pull_request":{"url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2229","id":51734513,"html_url":"https://github.com/ClusterHQ/flocker/pull/2229","diff_url":"https://github.com/ClusterHQ/flocker/pull/2229.diff","patch_url":"https://github.com/ClusterHQ/flocker/pull/2229.patch","issue_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2229","number":2229,"state":"open","locked":false,"title":"Update timeout logic to work with new flaky retries FLOC-3531","user":{"login":"Azulinho","id":851757,"avatar_url":"https://avatars.githubusercontent.com/u/851757?v=3","gravatar_id":"","url":"https://api.github.com/users/Azulinho","html_url":"https://github.com/Azulinho","followers_url":"https://api.github.com/users/Azulinho/followers","following_url":"https://api.github.com/users/Azulinho/following{/other_user}","gists_url":"https://api.github.com/users/Azulinho/gists{/gist_id}","starred_url":"https://api.github.com/users/Azulinho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Azulinho/subscriptions","organizations_url":"https://api.github.com/users/Azulinho/orgs","repos_url":"https://api.github.com/users/Azulinho/repos","events_url":"https://api.github.com/users/Azulinho/events{/privacy}","received_events_url":"https://api.github.com/users/Azulinho/received_events","type":"User","site_admin":false},"body":"https://clusterhq.atlassian.net/browse/FLOC-3531\r\n\r\nThe changes introduced by the flaky decorator increased the overal time of the acceptance tests as they retry failed steps.\r\nJenkins was configured to abort any jobs that took longer than 30minutes, as we assumed those jobs to be stuck.\r\nNow as the jobs take longer due to @flaky, normal running tests were being aborted by jenkins as they exceeded the 30 minutes window.\r\n\r\nThe initial goal of FLOC-3531 was to introduce an elastic timeout, where Jenkins would abort builds based on the average time taken over the last 'n' builds. \r\nThis data is however related to a single job, and we configure different jobs for each branch. \r\nThis means that any new PR branch being tested doesn't contain any valid data from previous builds.\r\nWe could still configure an elastic timeout with a default timeout for new jobs, but that would provide limited value for this particular usage case.\r\n\r\nInstead I am taking the shortcut (too much stuff on my hands) to simply address the long running jobs with an increased timeout, leaving all the short lived jobs with the same timeout settings.\r\n\r\nWe want to timeout a job as soon as possible in the event of that job being stuck, so that we can repurpose the jenkins slave builders back into the pool.\r\nBacklog trends from the jobs addressed show that these are good values to use at this moment.\r\n\r\nJenkins status here:\r\nhttp://ci-live.clusterhq.com:8080/job/ClusterHQ-flocker/job/update_timeout_logic_to_work_with_new_flaky_retries_FLOC-3531/\r\n\r\nnote that the 'admin' job has been failling for 11 days or so on master:\r\nhttp://ci-live.clusterhq.com:8080/job/ClusterHQ-flocker/job/master/job/run_trial_on_AWS_Ubuntu_Trusty_admin/buildTimeTrend\r\n\r\nsince:\r\n http://ci-live.clusterhq.com:8080/job/ClusterHQ-flocker/job/master/job/run_trial_on_AWS_Ubuntu_Trusty_admin/102/\r\n","created_at":"2015-11-24T23:48:32Z","updated_at":"2015-11-24T23:51:54Z","closed_at":null,"merged_at":null,"merge_commit_sha":"e28bfc63f56adfd545e3540710bb520653784193","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2229/commits","review_comments_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2229/comments","review_comment_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/comments{/number}","comments_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2229/comments","statuses_url":"https://api.github.com/repos/ClusterHQ/flocker/statuses/feb3e3d9ca966308ae4af4ccfc6aaf070d7c7677","head":{"label":"ClusterHQ:update_timeout_logic_to_work_with_new_flaky_retries_FLOC-3531","ref":"update_timeout_logic_to_work_with_new_flaky_retries_FLOC-3531","sha":"feb3e3d9ca966308ae4af4ccfc6aaf070d7c7677","user":{"login":"ClusterHQ","id":7913449,"avatar_url":"https://avatars.githubusercontent.com/u/7913449?v=3","gravatar_id":"","url":"https://api.github.com/users/ClusterHQ","html_url":"https://github.com/ClusterHQ","followers_url":"https://api.github.com/users/ClusterHQ/followers","following_url":"https://api.github.com/users/ClusterHQ/following{/other_user}","gists_url":"https://api.github.com/users/ClusterHQ/gists{/gist_id}","starred_url":"https://api.github.com/users/ClusterHQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClusterHQ/subscriptions","organizations_url":"https://api.github.com/users/ClusterHQ/orgs","repos_url":"https://api.github.com/users/ClusterHQ/repos","events_url":"https://api.github.com/users/ClusterHQ/events{/privacy}","received_events_url":"https://api.github.com/users/ClusterHQ/received_events","type":"Organization","site_admin":false},"repo":{"id":19248542,"name":"flocker","full_name":"ClusterHQ/flocker","owner":{"login":"ClusterHQ","id":7913449,"avatar_url":"https://avatars.githubusercontent.com/u/7913449?v=3","gravatar_id":"","url":"https://api.github.com/users/ClusterHQ","html_url":"https://github.com/ClusterHQ","followers_url":"https://api.github.com/users/ClusterHQ/followers","following_url":"https://api.github.com/users/ClusterHQ/following{/other_user}","gists_url":"https://api.github.com/users/ClusterHQ/gists{/gist_id}","starred_url":"https://api.github.com/users/ClusterHQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClusterHQ/subscriptions","organizations_url":"https://api.github.com/users/ClusterHQ/orgs","repos_url":"https://api.github.com/users/ClusterHQ/repos","events_url":"https://api.github.com/users/ClusterHQ/events{/privacy}","received_events_url":"https://api.github.com/users/ClusterHQ/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ClusterHQ/flocker","description":"Container data volume manager for your Dockerized application","fork":false,"url":"https://api.github.com/repos/ClusterHQ/flocker","forks_url":"https://api.github.com/repos/ClusterHQ/flocker/forks","keys_url":"https://api.github.com/repos/ClusterHQ/flocker/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ClusterHQ/flocker/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ClusterHQ/flocker/teams","hooks_url":"https://api.github.com/repos/ClusterHQ/flocker/hooks","issue_events_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/events{/number}","events_url":"https://api.github.com/repos/ClusterHQ/flocker/events","assignees_url":"https://api.github.com/repos/ClusterHQ/flocker/assignees{/user}","branches_url":"https://api.github.com/repos/ClusterHQ/flocker/branches{/branch}","tags_url":"https://api.github.com/repos/ClusterHQ/flocker/tags","blobs_url":"https://api.github.com/repos/ClusterHQ/flocker/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ClusterHQ/flocker/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ClusterHQ/flocker/git/refs{/sha}","trees_url":"https://api.github.com/repos/ClusterHQ/flocker/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ClusterHQ/flocker/statuses/{sha}","languages_url":"https://api.github.com/repos/ClusterHQ/flocker/languages","stargazers_url":"https://api.github.com/repos/ClusterHQ/flocker/stargazers","contributors_url":"https://api.github.com/repos/ClusterHQ/flocker/contributors","subscribers_url":"https://api.github.com/repos/ClusterHQ/flocker/subscribers","subscription_url":"https://api.github.com/repos/ClusterHQ/flocker/subscription","commits_url":"https://api.github.com/repos/ClusterHQ/flocker/commits{/sha}","git_commits_url":"https://api.github.com/repos/ClusterHQ/flocker/git/commits{/sha}","comments_url":"https://api.github.com/repos/ClusterHQ/flocker/comments{/number}","issue_comment_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/comments{/number}","contents_url":"https://api.github.com/repos/ClusterHQ/flocker/contents/{+path}","compare_url":"https://api.github.com/repos/ClusterHQ/flocker/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ClusterHQ/flocker/merges","archive_url":"https://api.github.com/repos/ClusterHQ/flocker/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ClusterHQ/flocker/downloads","issues_url":"https://api.github.com/repos/ClusterHQ/flocker/issues{/number}","pulls_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls{/number}","milestones_url":"https://api.github.com/repos/ClusterHQ/flocker/milestones{/number}","notifications_url":"https://api.github.com/repos/ClusterHQ/flocker/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ClusterHQ/flocker/labels{/name}","releases_url":"https://api.github.com/repos/ClusterHQ/flocker/releases{/id}","created_at":"2014-04-28T18:02:14Z","updated_at":"2015-11-24T21:25:17Z","pushed_at":"2015-11-24T23:48:32Z","git_url":"git://github.com/ClusterHQ/flocker.git","ssh_url":"git@github.com:ClusterHQ/flocker.git","clone_url":"https://github.com/ClusterHQ/flocker.git","svn_url":"https://github.com/ClusterHQ/flocker","homepage":"https://clusterhq.com","size":44877,"stargazers_count":1968,"watchers_count":1968,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":141,"mirror_url":null,"open_issues_count":32,"forks":141,"open_issues":32,"watchers":1968,"default_branch":"master"}},"base":{"label":"ClusterHQ:master","ref":"master","sha":"ea72a9df1a35f82bf5498dc6899ebf46d9864386","user":{"login":"ClusterHQ","id":7913449,"avatar_url":"https://avatars.githubusercontent.com/u/7913449?v=3","gravatar_id":"","url":"https://api.github.com/users/ClusterHQ","html_url":"https://github.com/ClusterHQ","followers_url":"https://api.github.com/users/ClusterHQ/followers","following_url":"https://api.github.com/users/ClusterHQ/following{/other_user}","gists_url":"https://api.github.com/users/ClusterHQ/gists{/gist_id}","starred_url":"https://api.github.com/users/ClusterHQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClusterHQ/subscriptions","organizations_url":"https://api.github.com/users/ClusterHQ/orgs","repos_url":"https://api.github.com/users/ClusterHQ/repos","events_url":"https://api.github.com/users/ClusterHQ/events{/privacy}","received_events_url":"https://api.github.com/users/ClusterHQ/received_events","type":"Organization","site_admin":false},"repo":{"id":19248542,"name":"flocker","full_name":"ClusterHQ/flocker","owner":{"login":"ClusterHQ","id":7913449,"avatar_url":"https://avatars.githubusercontent.com/u/7913449?v=3","gravatar_id":"","url":"https://api.github.com/users/ClusterHQ","html_url":"https://github.com/ClusterHQ","followers_url":"https://api.github.com/users/ClusterHQ/followers","following_url":"https://api.github.com/users/ClusterHQ/following{/other_user}","gists_url":"https://api.github.com/users/ClusterHQ/gists{/gist_id}","starred_url":"https://api.github.com/users/ClusterHQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClusterHQ/subscriptions","organizations_url":"https://api.github.com/users/ClusterHQ/orgs","repos_url":"https://api.github.com/users/ClusterHQ/repos","events_url":"https://api.github.com/users/ClusterHQ/events{/privacy}","received_events_url":"https://api.github.com/users/ClusterHQ/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ClusterHQ/flocker","description":"Container data volume manager for your Dockerized application","fork":false,"url":"https://api.github.com/repos/ClusterHQ/flocker","forks_url":"https://api.github.com/repos/ClusterHQ/flocker/forks","keys_url":"https://api.github.com/repos/ClusterHQ/flocker/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ClusterHQ/flocker/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ClusterHQ/flocker/teams","hooks_url":"https://api.github.com/repos/ClusterHQ/flocker/hooks","issue_events_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/events{/number}","events_url":"https://api.github.com/repos/ClusterHQ/flocker/events","assignees_url":"https://api.github.com/repos/ClusterHQ/flocker/assignees{/user}","branches_url":"https://api.github.com/repos/ClusterHQ/flocker/branches{/branch}","tags_url":"https://api.github.com/repos/ClusterHQ/flocker/tags","blobs_url":"https://api.github.com/repos/ClusterHQ/flocker/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ClusterHQ/flocker/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ClusterHQ/flocker/git/refs{/sha}","trees_url":"https://api.github.com/repos/ClusterHQ/flocker/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ClusterHQ/flocker/statuses/{sha}","languages_url":"https://api.github.com/repos/ClusterHQ/flocker/languages","stargazers_url":"https://api.github.com/repos/ClusterHQ/flocker/stargazers","contributors_url":"https://api.github.com/repos/ClusterHQ/flocker/contributors","subscribers_url":"https://api.github.com/repos/ClusterHQ/flocker/subscribers","subscription_url":"https://api.github.com/repos/ClusterHQ/flocker/subscription","commits_url":"https://api.github.com/repos/ClusterHQ/flocker/commits{/sha}","git_commits_url":"https://api.github.com/repos/ClusterHQ/flocker/git/commits{/sha}","comments_url":"https://api.github.com/repos/ClusterHQ/flocker/comments{/number}","issue_comment_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/comments{/number}","contents_url":"https://api.github.com/repos/ClusterHQ/flocker/contents/{+path}","compare_url":"https://api.github.com/repos/ClusterHQ/flocker/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ClusterHQ/flocker/merges","archive_url":"https://api.github.com/repos/ClusterHQ/flocker/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ClusterHQ/flocker/downloads","issues_url":"https://api.github.com/repos/ClusterHQ/flocker/issues{/number}","pulls_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls{/number}","milestones_url":"https://api.github.com/repos/ClusterHQ/flocker/milestones{/number}","notifications_url":"https://api.github.com/repos/ClusterHQ/flocker/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ClusterHQ/flocker/labels{/name}","releases_url":"https://api.github.com/repos/ClusterHQ/flocker/releases{/id}","created_at":"2014-04-28T18:02:14Z","updated_at":"2015-11-24T21:25:17Z","pushed_at":"2015-11-24T23:48:32Z","git_url":"git://github.com/ClusterHQ/flocker.git","ssh_url":"git@github.com:ClusterHQ/flocker.git","clone_url":"https://github.com/ClusterHQ/flocker.git","svn_url":"https://github.com/ClusterHQ/flocker","homepage":"https://clusterhq.com","size":44877,"stargazers_count":1968,"watchers_count":1968,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":141,"mirror_url":null,"open_issues_count":32,"forks":141,"open_issues":32,"watchers":1968,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2229"},"html":{"href":"https://github.com/ClusterHQ/flocker/pull/2229"},"issue":{"href":"https://api.github.com/repos/ClusterHQ/flocker/issues/2229"},"comments":{"href":"https://api.github.com/repos/ClusterHQ/flocker/issues/2229/comments"},"review_comments":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2229/comments"},"review_comment":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2229/commits"},"statuses":{"href":"https://api.github.com/repos/ClusterHQ/flocker/statuses/feb3e3d9ca966308ae4af4ccfc6aaf070d7c7677"}}}}