{"action":"opened","number":1005,"pull_request":{"url":"https://api.github.com/repos/openshift/origin/pulls/1005","id":29229319,"html_url":"https://github.com/openshift/origin/pull/1005","diff_url":"https://github.com/openshift/origin/pull/1005.diff","patch_url":"https://github.com/openshift/origin/pull/1005.patch","issue_url":"https://api.github.com/repos/openshift/origin/issues/1005","number":1005,"state":"open","locked":false,"title":"WIP: Replace deploy controller FIFOs with Store reconciliation","user":{"login":"ironcladlou","id":298299,"avatar_url":"https://avatars.githubusercontent.com/u/298299?v=3","gravatar_id":"","url":"https://api.github.com/users/ironcladlou","html_url":"https://github.com/ironcladlou","followers_url":"https://api.github.com/users/ironcladlou/followers","following_url":"https://api.github.com/users/ironcladlou/following{/other_user}","gists_url":"https://api.github.com/users/ironcladlou/gists{/gist_id}","starred_url":"https://api.github.com/users/ironcladlou/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ironcladlou/subscriptions","organizations_url":"https://api.github.com/users/ironcladlou/orgs","repos_url":"https://api.github.com/users/ironcladlou/repos","events_url":"https://api.github.com/users/ironcladlou/events{/privacy}","received_events_url":"https://api.github.com/users/ironcladlou/received_events","type":"User","site_admin":false},"body":"This PR is a POC to open discussion around replacing the use of `cache.FIFO` with state reconciliation loops over one or more `cache.Store`. It is an extension of upstream discussions ([#3521](https://github.com/GoogleCloudPlatform/kubernetes/pull/3521) and [#4330](https://github.com/GoogleCloudPlatform/kubernetes/issues/4330)) which pertain to the use of FIFO in the context of a state reconciling controller (such as the Kubernetes ReplicationController and Scheduler, and the Build and Deployment family of controllers in OpenShift). Our problems aren't unique.\r\n\r\nThe error handling/retry issues with our controllers described in #824 aren't easily resolved. There are some partial solutions upstream, and some other ideas proposed in [#4330](https://github.com/GoogleCloudPlatform/kubernetes/issues/4330), but they aren't proven and are added complexity.\r\n\r\nEven if retry handling were fully accounted for with FIFO, there are still outstanding issues with how to handle delete events in general (something the DeltaStore sought to partially address in [#3521](https://github.com/GoogleCloudPlatform/kubernetes/pull/3521)).\r\n\r\nGiven the lack of a unified solution which solves these problems in the context of FIFO use, and based on conversation with @pmorie, @derekwaynecarr, and upstream within [#4330](https://github.com/GoogleCloudPlatform/kubernetes/issues/4330) and [#3521](https://github.com/GoogleCloudPlatform/kubernetes/pull/3521), I suggest we port the controllers to a model which hinges on triggered full state reconciliation against `cache.Store`s (similar to how the ReplicationController currently works, but more flexible).\r\n\r\nThis PR contains a prototype of this methodology applied to the DeploymentConfigController. Some features:\r\n\r\n1. Sync triggering via channel allows for arbitrary, abstracted sync frequency: sync can be periodic, combined with watches, or any other criteria\r\n2. Resync triggering via a channel combined with abstracted error handling allows for external control of retry behavior: retries could result in an immediate resync, could incorporate backoff logic, etc.\r\n3. Missed events are handled naturally: full system state will eventually be be reconciled during the next interval\r\n4. Deletes become simple to handle via analysis of multiple stores at a single point in time\r\n\r\nThere are some obvious concerns about performance when introducing a linear iteration over unbounded sets, but also some benefits.\r\n\r\nCompared to the FIFO:\r\n\r\n1. In terms of max possible memory usage, use of a cache is no less worse than the `FIFO` at certain points in time in practice (consider the initial seeding of the FIFO with the Reflector, or a mass resource update, both of which result in all system resources being loaded into memory)\r\n\r\nThere are some interesting optimization opportunities with this model:\r\n\r\n1. Memory and parallelism optimization can be abstracted at a higher tier transparent to consumers (for example, partitioning data sets provided to individual controller caches in order to limit total memory consumption and distribute processing workload)\r\n2. Store abstraction allows for the possibility of implementing offloading/spooling data if necessary with no change to consumers\r\n\r\nThe net effect of this switch is a pattern which is far easier to reason about and which is more easily made correct for our event handling use cases, while keeping open the door to creative performance optimization.\r\n\r\nIf we can reach consensus on the pattern, my intent is to continue this work by revising the tests and formalizing this PR as model the next controllers should be refactored towards separately.\r\n","created_at":"2015-02-13T00:08:23Z","updated_at":"2015-02-13T00:08:23Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/openshift/origin/pulls/1005/commits","review_comments_url":"https://api.github.com/repos/openshift/origin/pulls/1005/comments","review_comment_url":"https://api.github.com/repos/openshift/origin/pulls/comments/{number}","comments_url":"https://api.github.com/repos/openshift/origin/issues/1005/comments","statuses_url":"https://api.github.com/repos/openshift/origin/statuses/eb75ed9ec711e3594976b9197089c952f1cdd73f","head":{"label":"ironcladlou:deploy-recon","ref":"deploy-recon","sha":"eb75ed9ec711e3594976b9197089c952f1cdd73f","user":{"login":"ironcladlou","id":298299,"avatar_url":"https://avatars.githubusercontent.com/u/298299?v=3","gravatar_id":"","url":"https://api.github.com/users/ironcladlou","html_url":"https://github.com/ironcladlou","followers_url":"https://api.github.com/users/ironcladlou/followers","following_url":"https://api.github.com/users/ironcladlou/following{/other_user}","gists_url":"https://api.github.com/users/ironcladlou/gists{/gist_id}","starred_url":"https://api.github.com/users/ironcladlou/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ironcladlou/subscriptions","organizations_url":"https://api.github.com/users/ironcladlou/orgs","repos_url":"https://api.github.com/users/ironcladlou/repos","events_url":"https://api.github.com/users/ironcladlou/events{/privacy}","received_events_url":"https://api.github.com/users/ironcladlou/received_events","type":"User","site_admin":false},"repo":{"id":23721352,"name":"origin","full_name":"ironcladlou/origin","owner":{"login":"ironcladlou","id":298299,"avatar_url":"https://avatars.githubusercontent.com/u/298299?v=3","gravatar_id":"","url":"https://api.github.com/users/ironcladlou","html_url":"https://github.com/ironcladlou","followers_url":"https://api.github.com/users/ironcladlou/followers","following_url":"https://api.github.com/users/ironcladlou/following{/other_user}","gists_url":"https://api.github.com/users/ironcladlou/gists{/gist_id}","starred_url":"https://api.github.com/users/ironcladlou/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ironcladlou/subscriptions","organizations_url":"https://api.github.com/users/ironcladlou/orgs","repos_url":"https://api.github.com/users/ironcladlou/repos","events_url":"https://api.github.com/users/ironcladlou/events{/privacy}","received_events_url":"https://api.github.com/users/ironcladlou/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ironcladlou/origin","description":"OpenShift 3 - build, deploy, and manage your applications with Docker and Kubernetes","fork":true,"url":"https://api.github.com/repos/ironcladlou/origin","forks_url":"https://api.github.com/repos/ironcladlou/origin/forks","keys_url":"https://api.github.com/repos/ironcladlou/origin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ironcladlou/origin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ironcladlou/origin/teams","hooks_url":"https://api.github.com/repos/ironcladlou/origin/hooks","issue_events_url":"https://api.github.com/repos/ironcladlou/origin/issues/events{/number}","events_url":"https://api.github.com/repos/ironcladlou/origin/events","assignees_url":"https://api.github.com/repos/ironcladlou/origin/assignees{/user}","branches_url":"https://api.github.com/repos/ironcladlou/origin/branches{/branch}","tags_url":"https://api.github.com/repos/ironcladlou/origin/tags","blobs_url":"https://api.github.com/repos/ironcladlou/origin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ironcladlou/origin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ironcladlou/origin/git/refs{/sha}","trees_url":"https://api.github.com/repos/ironcladlou/origin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ironcladlou/origin/statuses/{sha}","languages_url":"https://api.github.com/repos/ironcladlou/origin/languages","stargazers_url":"https://api.github.com/repos/ironcladlou/origin/stargazers","contributors_url":"https://api.github.com/repos/ironcladlou/origin/contributors","subscribers_url":"https://api.github.com/repos/ironcladlou/origin/subscribers","subscription_url":"https://api.github.com/repos/ironcladlou/origin/subscription","commits_url":"https://api.github.com/repos/ironcladlou/origin/commits{/sha}","git_commits_url":"https://api.github.com/repos/ironcladlou/origin/git/commits{/sha}","comments_url":"https://api.github.com/repos/ironcladlou/origin/comments{/number}","issue_comment_url":"https://api.github.com/repos/ironcladlou/origin/issues/comments/{number}","contents_url":"https://api.github.com/repos/ironcladlou/origin/contents/{+path}","compare_url":"https://api.github.com/repos/ironcladlou/origin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ironcladlou/origin/merges","archive_url":"https://api.github.com/repos/ironcladlou/origin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ironcladlou/origin/downloads","issues_url":"https://api.github.com/repos/ironcladlou/origin/issues{/number}","pulls_url":"https://api.github.com/repos/ironcladlou/origin/pulls{/number}","milestones_url":"https://api.github.com/repos/ironcladlou/origin/milestones{/number}","notifications_url":"https://api.github.com/repos/ironcladlou/origin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ironcladlou/origin/labels{/name}","releases_url":"https://api.github.com/repos/ironcladlou/origin/releases{/id}","created_at":"2014-09-05T23:22:09Z","updated_at":"2015-02-03T21:50:32Z","pushed_at":"2015-02-12T23:36:53Z","git_url":"git://github.com/ironcladlou/origin.git","ssh_url":"git@github.com:ironcladlou/origin.git","clone_url":"https://github.com/ironcladlou/origin.git","svn_url":"https://github.com/ironcladlou/origin","homepage":"https://openshift.github.io","size":39894,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"openshift:master","ref":"master","sha":"01f468d4ec4a293a3185c8bf6016b50e50d9465d","user":{"login":"openshift","id":792337,"avatar_url":"https://avatars.githubusercontent.com/u/792337?v=3","gravatar_id":"","url":"https://api.github.com/users/openshift","html_url":"https://github.com/openshift","followers_url":"https://api.github.com/users/openshift/followers","following_url":"https://api.github.com/users/openshift/following{/other_user}","gists_url":"https://api.github.com/users/openshift/gists{/gist_id}","starred_url":"https://api.github.com/users/openshift/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/openshift/subscriptions","organizations_url":"https://api.github.com/users/openshift/orgs","repos_url":"https://api.github.com/users/openshift/repos","events_url":"https://api.github.com/users/openshift/events{/privacy}","received_events_url":"https://api.github.com/users/openshift/received_events","type":"Organization","site_admin":false},"repo":{"id":22442668,"name":"origin","full_name":"openshift/origin","owner":{"login":"openshift","id":792337,"avatar_url":"https://avatars.githubusercontent.com/u/792337?v=3","gravatar_id":"","url":"https://api.github.com/users/openshift","html_url":"https://github.com/openshift","followers_url":"https://api.github.com/users/openshift/followers","following_url":"https://api.github.com/users/openshift/following{/other_user}","gists_url":"https://api.github.com/users/openshift/gists{/gist_id}","starred_url":"https://api.github.com/users/openshift/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/openshift/subscriptions","organizations_url":"https://api.github.com/users/openshift/orgs","repos_url":"https://api.github.com/users/openshift/repos","events_url":"https://api.github.com/users/openshift/events{/privacy}","received_events_url":"https://api.github.com/users/openshift/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/openshift/origin","description":"OpenShift 3 - build, deploy, and manage your applications with Docker and Kubernetes","fork":false,"url":"https://api.github.com/repos/openshift/origin","forks_url":"https://api.github.com/repos/openshift/origin/forks","keys_url":"https://api.github.com/repos/openshift/origin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/openshift/origin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/openshift/origin/teams","hooks_url":"https://api.github.com/repos/openshift/origin/hooks","issue_events_url":"https://api.github.com/repos/openshift/origin/issues/events{/number}","events_url":"https://api.github.com/repos/openshift/origin/events","assignees_url":"https://api.github.com/repos/openshift/origin/assignees{/user}","branches_url":"https://api.github.com/repos/openshift/origin/branches{/branch}","tags_url":"https://api.github.com/repos/openshift/origin/tags","blobs_url":"https://api.github.com/repos/openshift/origin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/openshift/origin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/openshift/origin/git/refs{/sha}","trees_url":"https://api.github.com/repos/openshift/origin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/openshift/origin/statuses/{sha}","languages_url":"https://api.github.com/repos/openshift/origin/languages","stargazers_url":"https://api.github.com/repos/openshift/origin/stargazers","contributors_url":"https://api.github.com/repos/openshift/origin/contributors","subscribers_url":"https://api.github.com/repos/openshift/origin/subscribers","subscription_url":"https://api.github.com/repos/openshift/origin/subscription","commits_url":"https://api.github.com/repos/openshift/origin/commits{/sha}","git_commits_url":"https://api.github.com/repos/openshift/origin/git/commits{/sha}","comments_url":"https://api.github.com/repos/openshift/origin/comments{/number}","issue_comment_url":"https://api.github.com/repos/openshift/origin/issues/comments/{number}","contents_url":"https://api.github.com/repos/openshift/origin/contents/{+path}","compare_url":"https://api.github.com/repos/openshift/origin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/openshift/origin/merges","archive_url":"https://api.github.com/repos/openshift/origin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/openshift/origin/downloads","issues_url":"https://api.github.com/repos/openshift/origin/issues{/number}","pulls_url":"https://api.github.com/repos/openshift/origin/pulls{/number}","milestones_url":"https://api.github.com/repos/openshift/origin/milestones{/number}","notifications_url":"https://api.github.com/repos/openshift/origin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/openshift/origin/labels{/name}","releases_url":"https://api.github.com/repos/openshift/origin/releases{/id}","created_at":"2014-07-30T20:04:05Z","updated_at":"2015-02-12T23:15:37Z","pushed_at":"2015-02-12T23:15:36Z","git_url":"git://github.com/openshift/origin.git","ssh_url":"git@github.com:openshift/origin.git","clone_url":"https://github.com/openshift/origin.git","svn_url":"https://github.com/openshift/origin","homepage":"https://openshift.github.io","size":54010,"stargazers_count":280,"watchers_count":280,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":148,"mirror_url":null,"open_issues_count":76,"forks":148,"open_issues":76,"watchers":280,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/openshift/origin/pulls/1005"},"html":{"href":"https://github.com/openshift/origin/pull/1005"},"issue":{"href":"https://api.github.com/repos/openshift/origin/issues/1005"},"comments":{"href":"https://api.github.com/repos/openshift/origin/issues/1005/comments"},"review_comments":{"href":"https://api.github.com/repos/openshift/origin/pulls/1005/comments"},"review_comment":{"href":"https://api.github.com/repos/openshift/origin/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/openshift/origin/pulls/1005/commits"},"statuses":{"href":"https://api.github.com/repos/openshift/origin/statuses/eb75ed9ec711e3594976b9197089c952f1cdd73f"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":1,"additions":178,"deletions":74,"changed_files":4}}