{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18162","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18162/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18162/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18162/events","html_url":"https://github.com/kubernetes/kubernetes/issues/18162","id":120239365,"number":18162,"title":"Modify conformance test policy to use tests from release branches","user":{"login":"spiffxp","id":49258,"avatar_url":"https://avatars.githubusercontent.com/u/49258?v=3","gravatar_id":"","url":"https://api.github.com/users/spiffxp","html_url":"https://github.com/spiffxp","followers_url":"https://api.github.com/users/spiffxp/followers","following_url":"https://api.github.com/users/spiffxp/following{/other_user}","gists_url":"https://api.github.com/users/spiffxp/gists{/gist_id}","starred_url":"https://api.github.com/users/spiffxp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spiffxp/subscriptions","organizations_url":"https://api.github.com/users/spiffxp/orgs","repos_url":"https://api.github.com/users/spiffxp/repos","events_url":"https://api.github.com/users/spiffxp/events{/privacy}","received_events_url":"https://api.github.com/users/spiffxp/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/test-infra","name":"team/test-infra","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"ihmccreery","id":368416,"avatar_url":"https://avatars.githubusercontent.com/u/368416?v=3","gravatar_id":"","url":"https://api.github.com/users/ihmccreery","html_url":"https://github.com/ihmccreery","followers_url":"https://api.github.com/users/ihmccreery/followers","following_url":"https://api.github.com/users/ihmccreery/following{/other_user}","gists_url":"https://api.github.com/users/ihmccreery/gists{/gist_id}","starred_url":"https://api.github.com/users/ihmccreery/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ihmccreery/subscriptions","organizations_url":"https://api.github.com/users/ihmccreery/orgs","repos_url":"https://api.github.com/users/ihmccreery/repos","events_url":"https://api.github.com/users/ihmccreery/events{/privacy}","received_events_url":"https://api.github.com/users/ihmccreery/received_events","type":"User","site_admin":false},"milestone":null,"comments":6,"created_at":"2015-12-03T18:35:24Z","updated_at":"2015-12-03T23:38:03Z","closed_at":null,"body":"Currently conformance testing uses the 'conformance-test-v1' branch.  We've run into issues where this branch doesn't get as actively maintained as the release branches.  This is an umbrella issue to cover migrating to a new conformance test policy.  See discussion in #17536 for more details\r\n\r\nNew conformance tests policy: tests should be run off of the release branch that corresponds to the level of conformance being tested, eg:\r\n- the release-1.0 branch should be used to test that a given cluster conforms up to v1.0\r\n- the release-1.1 branch should be used to test that a given cluster conforms up to v1.1\r\n- conformance tests are not guaranteed to be stable on the master branch, but stabilized once a release-X.Y branch is created\r\n\r\nProposed steps\r\n- [ ] update documentation for conformance testing (#14913)\r\n- [ ] update conformance test scripts to work according to the new policy\r\n- [ ] ensure conformance tests pass according to the new policy, bug fix / cherry-pick as appropriate\r\n  - [ ] release-1.1 branch passes against a v1.1 cluster on AWS\r\n  - [ ] release-1.1 branch passes against a v1.1 cluster on GCE\r\n  - [ ] release-1.0 branch passes against a v1.0 cluster on AWS\r\n  - [ ] release-1.0 branch passes against a v1.0 cluster on GCE\r\n  - [ ] release-1.0 branch passes against a v1.1 cluster on AWS\r\n  - [ ] release-1.0 branch passes against a v1.1 cluster on GCE\r\n- [ ] ? update CI scripts as appropriate\r\n- [ ] cleanup\r\n  - [ ] remove conformance-test-v1 branch (#17536)\r\n  - [ ] close out #16860\r\n  - [ ] close out #12014\r\n\r\nrelated, #12941"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/161825094","html_url":"https://github.com/kubernetes/kubernetes/issues/18162#issuecomment-161825094","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18162","id":161825094,"user":{"login":"spiffxp","id":49258,"avatar_url":"https://avatars.githubusercontent.com/u/49258?v=3","gravatar_id":"","url":"https://api.github.com/users/spiffxp","html_url":"https://github.com/spiffxp","followers_url":"https://api.github.com/users/spiffxp/followers","following_url":"https://api.github.com/users/spiffxp/following{/other_user}","gists_url":"https://api.github.com/users/spiffxp/gists{/gist_id}","starred_url":"https://api.github.com/users/spiffxp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spiffxp/subscriptions","organizations_url":"https://api.github.com/users/spiffxp/orgs","repos_url":"https://api.github.com/users/spiffxp/repos","events_url":"https://api.github.com/users/spiffxp/events{/privacy}","received_events_url":"https://api.github.com/users/spiffxp/received_events","type":"User","site_admin":false},"created_at":"2015-12-03T23:38:03Z","updated_at":"2015-12-03T23:38:03Z","body":">That sounds like a recipe for scrambling when we branch and realize we broke v1.0 conformance two months ago on master.\r\n\r\nTrue.  Maybe I'm overestimating the add'l overhead this would add.  Just thinking down the line when we have N releases, it may be expensive / slow to do N compatibility checks on every commit.  Since master is in such high flux, capture that with a regularly scheduled job rather than gate per-commit.  But do it per-commit for the release branches.\r\n\r\n>Per versioning.md, branching and cutting the beta version happen at the same time.\r\n\r\nWould mandating that conformance tests pass prior to cutting alpha releases off of master make sense then?  I could see that imposing too much of a code-freeze mentality on master.  Alternatively, we focus on stabilizing conformance tests after the first beta release is cut, prior to the first non-beta release."}}