{"action":"created","comment":{"url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/comments/40782085","id":40782085,"diff_hunk":"@@ -455,27 +458,52 @@ def _send_state_to_connections(self, connections):\n         \"\"\"\n         configuration = self.configuration_service.get()\n         state = self.cluster_state.as_deployment()\n-        with LOG_SEND_CLUSTER_STATE(self.logger,\n-                                    configuration=configuration,\n+        with LOG_SEND_CLUSTER_STATE(configuration=configuration,\n                                     state=state):\n             with _caching_encoder.cache():\n                 for connection in connections:\n-                    action = LOG_SEND_TO_AGENT(self.logger, agent=connection)\n-                    with action.context():\n-                        # XXX If callRemote raises an exception, the loop won't\n-                        # finish and the rest of the connections won't receive\n-                        # the updated state.  Asynchronous exceptions aren't a\n-                        # problem since they won't interrupt the loop (and they\n-                        # shouldn't be allowed to).  No test coverage for\n-                        # either of these cases.\n-                        d = DeferredContext(connection.callRemote(\n-                            ClusterStatusCommand,\n-                            configuration=configuration,\n-                            state=state,\n-                            eliot_context=action\n-                        ))\n-                        d.addActionFinish()\n-                        d.result.addErrback(lambda _: None)\n+                    # XXX If callRemote raises an exception, the loop won't","path":"flocker/control/_protocol.py","position":82,"original_position":82,"commit_id":"f2ada3c79c64c620989701d82b5ee280e965098e","original_commit_id":"f2ada3c79c64c620989701d82b5ee280e965098e","user":{"login":"dwgebler","id":646585,"avatar_url":"https://avatars.githubusercontent.com/u/646585?v=3","gravatar_id":"","url":"https://api.github.com/users/dwgebler","html_url":"https://github.com/dwgebler","followers_url":"https://api.github.com/users/dwgebler/followers","following_url":"https://api.github.com/users/dwgebler/following{/other_user}","gists_url":"https://api.github.com/users/dwgebler/gists{/gist_id}","starred_url":"https://api.github.com/users/dwgebler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dwgebler/subscriptions","organizations_url":"https://api.github.com/users/dwgebler/orgs","repos_url":"https://api.github.com/users/dwgebler/repos","events_url":"https://api.github.com/users/dwgebler/events{/privacy}","received_events_url":"https://api.github.com/users/dwgebler/received_events","type":"User","site_admin":false},"body":"This comment looks like it belongs in `_update_connection` now.","created_at":"2015-09-30T11:10:31Z","updated_at":"2015-09-30T11:10:31Z","html_url":"https://github.com/ClusterHQ/flocker/pull/2002#discussion_r40782085","pull_request_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2002","_links":{"self":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/comments/40782085"},"html":{"href":"https://github.com/ClusterHQ/flocker/pull/2002#discussion_r40782085"},"pull_request":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2002"}}},"pull_request":{"url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2002","id":46349398,"html_url":"https://github.com/ClusterHQ/flocker/pull/2002","diff_url":"https://github.com/ClusterHQ/flocker/pull/2002.diff","patch_url":"https://github.com/ClusterHQ/flocker/pull/2002.patch","issue_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2002","number":2002,"state":"open","locked":false,"title":"[FLOC-3140] Coalesce redundant cluster parameter updates to agents","user":{"login":"exarkun","id":254565,"avatar_url":"https://avatars.githubusercontent.com/u/254565?v=3","gravatar_id":"","url":"https://api.github.com/users/exarkun","html_url":"https://github.com/exarkun","followers_url":"https://api.github.com/users/exarkun/followers","following_url":"https://api.github.com/users/exarkun/following{/other_user}","gists_url":"https://api.github.com/users/exarkun/gists{/gist_id}","starred_url":"https://api.github.com/users/exarkun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/exarkun/subscriptions","organizations_url":"https://api.github.com/users/exarkun/orgs","repos_url":"https://api.github.com/users/exarkun/repos","events_url":"https://api.github.com/users/exarkun/events{/privacy}","received_events_url":"https://api.github.com/users/exarkun/received_events","type":"User","site_admin":false},"body":"Fixes https://clusterhq.atlassian.net/browse/FLOC-3140\r\n\r\nThe general idea here is that because many different things trigger a full update of cluster parameters (state and configuration) to agents, it is often the case that several full updates will be sent to an agent before it has even had a chance to process the first update.  In these cases, the first update is unavoidable and the last update is useful but the intermediate updates are useless and we can avoid sending them.  So, do.\r\n\r\nSince we only have manual, ad hoc benchmarking in place at the moment, here are some results from my manual, ad hoc benchmarks:\r\n\r\nWithout this change, it takes longer than 10 minutes (I didn't wait to see much much longer) for flocker-control to finish processing all of the events generated on the startup of a 40 node cluster.\r\n\r\nWith this change, it takes about 3 minutes.\r\n\r\nWith this change, on an empty 40 node cluster it takes slightly less than 6 minutes to configure 593 containers (out of an attempted 800, the difference is accounted for by errors from the API server).  Measuring from when the first container is configured, it takes a little less than 9 minutes to converge on this configuration.  While convergence is underway, it takes a little under 2 minutes for the control service to respond to a read-only API request.  Once the cluster has converged, it takes 14 seconds to add one more container to the configuration and an additional 16 seconds to converge on that configuration.\r\n\r\nAs a follow-up to this, I've made an additional change which cuts down the cost of some Eliot logging near the code changed by this PR.  That involved some refactoring which somewhat improves the readability of this code.  I didn't have a chance this evening to push that refactoring into this PR so what's here is a bit of a mess.  Please have a look at https://github.com/ClusterHQ/flocker/compare/coalesce-calls-probably-even-better-FLOC-3140...send_cluster_state_logging-FLOC-3151?expand=1 to see the improved factoring and don't count the mess in this PR as a reason not to accept this change (since it is required before send_cluster_state_logging-FLOC-3151 can be merged).\r\n\r\nAs far as I know, send_cluster_state_logging-FLOC-3151 is also ready for a PR but since it's stacked on this branch github will make a mess of the diff - so I'm aiming for this to be merged before creating that PR.\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/clusterhq/flocker/2002)\n<!-- Reviewable:end -->\n","created_at":"2015-09-30T00:29:37Z","updated_at":"2015-09-30T11:10:31Z","closed_at":null,"merged_at":null,"merge_commit_sha":"219f846d7b4dbf8c8d0fa903025922dbcbbd055b","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2002/commits","review_comments_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2002/comments","review_comment_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/comments{/number}","comments_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2002/comments","statuses_url":"https://api.github.com/repos/ClusterHQ/flocker/statuses/f2ada3c79c64c620989701d82b5ee280e965098e","head":{"label":"ClusterHQ:coalesce-calls-probably-even-better-FLOC-3140","ref":"coalesce-calls-probably-even-better-FLOC-3140","sha":"f2ada3c79c64c620989701d82b5ee280e965098e","user":{"login":"ClusterHQ","id":7913449,"avatar_url":"https://avatars.githubusercontent.com/u/7913449?v=3","gravatar_id":"","url":"https://api.github.com/users/ClusterHQ","html_url":"https://github.com/ClusterHQ","followers_url":"https://api.github.com/users/ClusterHQ/followers","following_url":"https://api.github.com/users/ClusterHQ/following{/other_user}","gists_url":"https://api.github.com/users/ClusterHQ/gists{/gist_id}","starred_url":"https://api.github.com/users/ClusterHQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClusterHQ/subscriptions","organizations_url":"https://api.github.com/users/ClusterHQ/orgs","repos_url":"https://api.github.com/users/ClusterHQ/repos","events_url":"https://api.github.com/users/ClusterHQ/events{/privacy}","received_events_url":"https://api.github.com/users/ClusterHQ/received_events","type":"Organization","site_admin":false},"repo":{"id":19248542,"name":"flocker","full_name":"ClusterHQ/flocker","owner":{"login":"ClusterHQ","id":7913449,"avatar_url":"https://avatars.githubusercontent.com/u/7913449?v=3","gravatar_id":"","url":"https://api.github.com/users/ClusterHQ","html_url":"https://github.com/ClusterHQ","followers_url":"https://api.github.com/users/ClusterHQ/followers","following_url":"https://api.github.com/users/ClusterHQ/following{/other_user}","gists_url":"https://api.github.com/users/ClusterHQ/gists{/gist_id}","starred_url":"https://api.github.com/users/ClusterHQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClusterHQ/subscriptions","organizations_url":"https://api.github.com/users/ClusterHQ/orgs","repos_url":"https://api.github.com/users/ClusterHQ/repos","events_url":"https://api.github.com/users/ClusterHQ/events{/privacy}","received_events_url":"https://api.github.com/users/ClusterHQ/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ClusterHQ/flocker","description":"Easily manage Docker containers & their data","fork":false,"url":"https://api.github.com/repos/ClusterHQ/flocker","forks_url":"https://api.github.com/repos/ClusterHQ/flocker/forks","keys_url":"https://api.github.com/repos/ClusterHQ/flocker/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ClusterHQ/flocker/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ClusterHQ/flocker/teams","hooks_url":"https://api.github.com/repos/ClusterHQ/flocker/hooks","issue_events_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/events{/number}","events_url":"https://api.github.com/repos/ClusterHQ/flocker/events","assignees_url":"https://api.github.com/repos/ClusterHQ/flocker/assignees{/user}","branches_url":"https://api.github.com/repos/ClusterHQ/flocker/branches{/branch}","tags_url":"https://api.github.com/repos/ClusterHQ/flocker/tags","blobs_url":"https://api.github.com/repos/ClusterHQ/flocker/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ClusterHQ/flocker/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ClusterHQ/flocker/git/refs{/sha}","trees_url":"https://api.github.com/repos/ClusterHQ/flocker/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ClusterHQ/flocker/statuses/{sha}","languages_url":"https://api.github.com/repos/ClusterHQ/flocker/languages","stargazers_url":"https://api.github.com/repos/ClusterHQ/flocker/stargazers","contributors_url":"https://api.github.com/repos/ClusterHQ/flocker/contributors","subscribers_url":"https://api.github.com/repos/ClusterHQ/flocker/subscribers","subscription_url":"https://api.github.com/repos/ClusterHQ/flocker/subscription","commits_url":"https://api.github.com/repos/ClusterHQ/flocker/commits{/sha}","git_commits_url":"https://api.github.com/repos/ClusterHQ/flocker/git/commits{/sha}","comments_url":"https://api.github.com/repos/ClusterHQ/flocker/comments{/number}","issue_comment_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/comments{/number}","contents_url":"https://api.github.com/repos/ClusterHQ/flocker/contents/{+path}","compare_url":"https://api.github.com/repos/ClusterHQ/flocker/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ClusterHQ/flocker/merges","archive_url":"https://api.github.com/repos/ClusterHQ/flocker/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ClusterHQ/flocker/downloads","issues_url":"https://api.github.com/repos/ClusterHQ/flocker/issues{/number}","pulls_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls{/number}","milestones_url":"https://api.github.com/repos/ClusterHQ/flocker/milestones{/number}","notifications_url":"https://api.github.com/repos/ClusterHQ/flocker/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ClusterHQ/flocker/labels{/name}","releases_url":"https://api.github.com/repos/ClusterHQ/flocker/releases{/id}","created_at":"2014-04-28T18:02:14Z","updated_at":"2015-09-28T17:40:30Z","pushed_at":"2015-09-30T10:43:43Z","git_url":"git://github.com/ClusterHQ/flocker.git","ssh_url":"git@github.com:ClusterHQ/flocker.git","clone_url":"https://github.com/ClusterHQ/flocker.git","svn_url":"https://github.com/ClusterHQ/flocker","homepage":"https://clusterhq.com","size":68044,"stargazers_count":1876,"watchers_count":1876,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":133,"mirror_url":null,"open_issues_count":33,"forks":133,"open_issues":33,"watchers":1876,"default_branch":"master"}},"base":{"label":"ClusterHQ:master","ref":"master","sha":"7edbd00d631f6f6f79018349f3a0f802eedb81ea","user":{"login":"ClusterHQ","id":7913449,"avatar_url":"https://avatars.githubusercontent.com/u/7913449?v=3","gravatar_id":"","url":"https://api.github.com/users/ClusterHQ","html_url":"https://github.com/ClusterHQ","followers_url":"https://api.github.com/users/ClusterHQ/followers","following_url":"https://api.github.com/users/ClusterHQ/following{/other_user}","gists_url":"https://api.github.com/users/ClusterHQ/gists{/gist_id}","starred_url":"https://api.github.com/users/ClusterHQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClusterHQ/subscriptions","organizations_url":"https://api.github.com/users/ClusterHQ/orgs","repos_url":"https://api.github.com/users/ClusterHQ/repos","events_url":"https://api.github.com/users/ClusterHQ/events{/privacy}","received_events_url":"https://api.github.com/users/ClusterHQ/received_events","type":"Organization","site_admin":false},"repo":{"id":19248542,"name":"flocker","full_name":"ClusterHQ/flocker","owner":{"login":"ClusterHQ","id":7913449,"avatar_url":"https://avatars.githubusercontent.com/u/7913449?v=3","gravatar_id":"","url":"https://api.github.com/users/ClusterHQ","html_url":"https://github.com/ClusterHQ","followers_url":"https://api.github.com/users/ClusterHQ/followers","following_url":"https://api.github.com/users/ClusterHQ/following{/other_user}","gists_url":"https://api.github.com/users/ClusterHQ/gists{/gist_id}","starred_url":"https://api.github.com/users/ClusterHQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClusterHQ/subscriptions","organizations_url":"https://api.github.com/users/ClusterHQ/orgs","repos_url":"https://api.github.com/users/ClusterHQ/repos","events_url":"https://api.github.com/users/ClusterHQ/events{/privacy}","received_events_url":"https://api.github.com/users/ClusterHQ/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ClusterHQ/flocker","description":"Easily manage Docker containers & their data","fork":false,"url":"https://api.github.com/repos/ClusterHQ/flocker","forks_url":"https://api.github.com/repos/ClusterHQ/flocker/forks","keys_url":"https://api.github.com/repos/ClusterHQ/flocker/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ClusterHQ/flocker/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ClusterHQ/flocker/teams","hooks_url":"https://api.github.com/repos/ClusterHQ/flocker/hooks","issue_events_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/events{/number}","events_url":"https://api.github.com/repos/ClusterHQ/flocker/events","assignees_url":"https://api.github.com/repos/ClusterHQ/flocker/assignees{/user}","branches_url":"https://api.github.com/repos/ClusterHQ/flocker/branches{/branch}","tags_url":"https://api.github.com/repos/ClusterHQ/flocker/tags","blobs_url":"https://api.github.com/repos/ClusterHQ/flocker/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ClusterHQ/flocker/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ClusterHQ/flocker/git/refs{/sha}","trees_url":"https://api.github.com/repos/ClusterHQ/flocker/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ClusterHQ/flocker/statuses/{sha}","languages_url":"https://api.github.com/repos/ClusterHQ/flocker/languages","stargazers_url":"https://api.github.com/repos/ClusterHQ/flocker/stargazers","contributors_url":"https://api.github.com/repos/ClusterHQ/flocker/contributors","subscribers_url":"https://api.github.com/repos/ClusterHQ/flocker/subscribers","subscription_url":"https://api.github.com/repos/ClusterHQ/flocker/subscription","commits_url":"https://api.github.com/repos/ClusterHQ/flocker/commits{/sha}","git_commits_url":"https://api.github.com/repos/ClusterHQ/flocker/git/commits{/sha}","comments_url":"https://api.github.com/repos/ClusterHQ/flocker/comments{/number}","issue_comment_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/comments{/number}","contents_url":"https://api.github.com/repos/ClusterHQ/flocker/contents/{+path}","compare_url":"https://api.github.com/repos/ClusterHQ/flocker/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ClusterHQ/flocker/merges","archive_url":"https://api.github.com/repos/ClusterHQ/flocker/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ClusterHQ/flocker/downloads","issues_url":"https://api.github.com/repos/ClusterHQ/flocker/issues{/number}","pulls_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls{/number}","milestones_url":"https://api.github.com/repos/ClusterHQ/flocker/milestones{/number}","notifications_url":"https://api.github.com/repos/ClusterHQ/flocker/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ClusterHQ/flocker/labels{/name}","releases_url":"https://api.github.com/repos/ClusterHQ/flocker/releases{/id}","created_at":"2014-04-28T18:02:14Z","updated_at":"2015-09-28T17:40:30Z","pushed_at":"2015-09-30T10:43:43Z","git_url":"git://github.com/ClusterHQ/flocker.git","ssh_url":"git@github.com:ClusterHQ/flocker.git","clone_url":"https://github.com/ClusterHQ/flocker.git","svn_url":"https://github.com/ClusterHQ/flocker","homepage":"https://clusterhq.com","size":68044,"stargazers_count":1876,"watchers_count":1876,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":133,"mirror_url":null,"open_issues_count":33,"forks":133,"open_issues":33,"watchers":1876,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2002"},"html":{"href":"https://github.com/ClusterHQ/flocker/pull/2002"},"issue":{"href":"https://api.github.com/repos/ClusterHQ/flocker/issues/2002"},"comments":{"href":"https://api.github.com/repos/ClusterHQ/flocker/issues/2002/comments"},"review_comments":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2002/comments"},"review_comment":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2002/commits"},"statuses":{"href":"https://api.github.com/repos/ClusterHQ/flocker/statuses/f2ada3c79c64c620989701d82b5ee280e965098e"}}}}