{"action":"created","comment":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch/pulls/comments/24799776","id":24799776,"diff_hunk":"@@ -596,6 +588,90 @@ public boolean apply(Object input) {\n     }\n \n     /**\n+     * Tests that emulates a frozen elected master node that unfreezes and pushes his cluster state to other nodes\n+     * that already are following another elected master node. These nodes should reject this cluster state and prevent\n+     * them from following the stale master.\n+     */\n+    @Test\n+    public void testStaleMasterNotHijackingMajority() throws Exception {\n+        // TODO: on mac OS multicast threads are shared between nodes and we therefore we can't simulate GC and stop pinging for just one node\n+        // find a way to block thread creation in the generic thread pool to avoid this.\n+        // 3 node cluster with unicast discovery and minimum_master_nodes set to 2:\n+        List<String> nodes = startUnicastCluster(3, null, 2);\n+\n+        // Save the current master node as old master node, because that node will get frozen\n+        final String oldMasterNode = internalCluster().getMasterName();\n+        for (String node : nodes) {\n+            client(node).admin().cluster().prepareHealth().setWaitForEvents(Priority.LANGUID).setLocal(true).get();\n+            ensureStableCluster(3, node);\n+        }\n+        assertMaster(oldMasterNode, nodes);\n+\n+        // Simulating a painful gc by suspending all threads for a long time on the current elected master node.\n+        SingleNodeDisruption masterNodeDisruption = new LongGCDisruption(getRandom(), oldMasterNode);\n+\n+        // Save the majority side\n+        final List<String> majoritySide = new ArrayList<>(nodes);\n+        majoritySide.remove(oldMasterNode);\n+\n+        // Keep track of the masters that appear in the cluster state on both nodes on the majority side,\n+        // only new_master may appear in here.\n+        final Set<String> masters = new HashSet<>();\n+        for (String node : majoritySide) {\n+            internalCluster().getInstance(ClusterService.class, node).add(new ClusterStateListener() {\n+                @Override\n+                public void clusterChanged(ClusterChangedEvent event) {\n+                    DiscoveryNode masterNode = event.state().getNodes().masterNode();","path":"src/test/java/org/elasticsearch/discovery/DiscoveryWithServiceDisruptions.java","position":72,"original_position":72,"commit_id":"d77f74c0fde416cf613972a7bbdd84b4c0e48a88","original_commit_id":"d77f74c0fde416cf613972a7bbdd84b4c0e48a88","user":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"body":"can we track changes here instead of master nodes? I think it would be interesting to test we don't have  newMaster -> null -> newMaster elected again situation. Something ala (pseudo code :)):\r\n\r\n```\r\nfinal Map<String,List<Tuple<String,String>>> masterChangePerNode\r\n```\r\nand then:\r\n\r\n```\r\nif (event.previousState.masterNodeId != event.state.masterNodeId) { \r\n  synchronize(masterChangePerNode) {\r\n      masterChangePerNode.addToNodeId(new Tuple<oldMasterId,newMasterId>)\r\n }\r\n```\r\n","created_at":"2015-02-17T08:35:13Z","updated_at":"2015-02-17T08:35:13Z","html_url":"https://github.com/elasticsearch/elasticsearch/pull/9632#discussion_r24799776","pull_request_url":"https://api.github.com/repos/elasticsearch/elasticsearch/pulls/9632","_links":{"self":{"href":"https://api.github.com/repos/elasticsearch/elasticsearch/pulls/comments/24799776"},"html":{"href":"https://github.com/elasticsearch/elasticsearch/pull/9632#discussion_r24799776"},"pull_request":{"href":"https://api.github.com/repos/elasticsearch/elasticsearch/pulls/9632"}}},"pull_request":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch/pulls/9632","id":29015034,"html_url":"https://github.com/elasticsearch/elasticsearch/pull/9632","diff_url":"https://github.com/elasticsearch/elasticsearch/pull/9632.diff","patch_url":"https://github.com/elasticsearch/elasticsearch/pull/9632.patch","issue_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9632","number":9632,"state":"open","locked":false,"title":"[Discovery] Prevent stale master nodes from sharing dated cluster states to nodes that have moved to a different master node","user":{"login":"martijnvg","id":580421,"avatar_url":"https://avatars.githubusercontent.com/u/580421?v=3","gravatar_id":"","url":"https://api.github.com/users/martijnvg","html_url":"https://github.com/martijnvg","followers_url":"https://api.github.com/users/martijnvg/followers","following_url":"https://api.github.com/users/martijnvg/following{/other_user}","gists_url":"https://api.github.com/users/martijnvg/gists{/gist_id}","starred_url":"https://api.github.com/users/martijnvg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martijnvg/subscriptions","organizations_url":"https://api.github.com/users/martijnvg/orgs","repos_url":"https://api.github.com/users/martijnvg/repos","events_url":"https://api.github.com/users/martijnvg/events{/privacy}","received_events_url":"https://api.github.com/users/martijnvg/received_events","type":"User","site_admin":false},"body":"If an elected master node goes into a long gc then other nodes' fault detection will notice this and a new master election is started and eventually a new master node is elected. If the previous master nodes goes out of the long gc it can still have pending tasks which can result in new cluster state updates. Nodes that are still in the nodes list of this previous elected master node can get these cluster state updates. This commit makes sure that this dated cluster states are not accepted by these nodes.\r\n\r\nThis issue can temporary lead to the fact that non elected master nodes switch to the previous elected master node. The new elected master node also gets the same dated cluster state, but rejects it and tells the previous elected master node to step down and rejoin. Because the new elected master is the only master node the previous elected master node will follow the new elected master node. Any nodes that follow the previous elected master node (by accident), will also rejoin and follow the new elected master node because their master fault detection will fail. So all in all this isn't a severe problem, because the problem fixes itself eventually.","created_at":"2015-02-10T15:20:21Z","updated_at":"2015-02-17T08:35:13Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b60810286623a9227863a7834d210741c636dfae","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/elasticsearch/elasticsearch/pulls/9632/commits","review_comments_url":"https://api.github.com/repos/elasticsearch/elasticsearch/pulls/9632/comments","review_comment_url":"https://api.github.com/repos/elasticsearch/elasticsearch/pulls/comments/{number}","comments_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9632/comments","statuses_url":"https://api.github.com/repos/elasticsearch/elasticsearch/statuses/d77f74c0fde416cf613972a7bbdd84b4c0e48a88","head":{"label":"martijnvg:zen/prevent-master-node-hijacking","ref":"zen/prevent-master-node-hijacking","sha":"d77f74c0fde416cf613972a7bbdd84b4c0e48a88","user":{"login":"martijnvg","id":580421,"avatar_url":"https://avatars.githubusercontent.com/u/580421?v=3","gravatar_id":"","url":"https://api.github.com/users/martijnvg","html_url":"https://github.com/martijnvg","followers_url":"https://api.github.com/users/martijnvg/followers","following_url":"https://api.github.com/users/martijnvg/following{/other_user}","gists_url":"https://api.github.com/users/martijnvg/gists{/gist_id}","starred_url":"https://api.github.com/users/martijnvg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martijnvg/subscriptions","organizations_url":"https://api.github.com/users/martijnvg/orgs","repos_url":"https://api.github.com/users/martijnvg/repos","events_url":"https://api.github.com/users/martijnvg/events{/privacy}","received_events_url":"https://api.github.com/users/martijnvg/received_events","type":"User","site_admin":false},"repo":{"id":3584739,"name":"elasticsearch","full_name":"martijnvg/elasticsearch","owner":{"login":"martijnvg","id":580421,"avatar_url":"https://avatars.githubusercontent.com/u/580421?v=3","gravatar_id":"","url":"https://api.github.com/users/martijnvg","html_url":"https://github.com/martijnvg","followers_url":"https://api.github.com/users/martijnvg/followers","following_url":"https://api.github.com/users/martijnvg/following{/other_user}","gists_url":"https://api.github.com/users/martijnvg/gists{/gist_id}","starred_url":"https://api.github.com/users/martijnvg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martijnvg/subscriptions","organizations_url":"https://api.github.com/users/martijnvg/orgs","repos_url":"https://api.github.com/users/martijnvg/repos","events_url":"https://api.github.com/users/martijnvg/events{/privacy}","received_events_url":"https://api.github.com/users/martijnvg/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/martijnvg/elasticsearch","description":"Open Source, Distributed, RESTful Search Engine with grouping","fork":true,"url":"https://api.github.com/repos/martijnvg/elasticsearch","forks_url":"https://api.github.com/repos/martijnvg/elasticsearch/forks","keys_url":"https://api.github.com/repos/martijnvg/elasticsearch/keys{/key_id}","collaborators_url":"https://api.github.com/repos/martijnvg/elasticsearch/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/martijnvg/elasticsearch/teams","hooks_url":"https://api.github.com/repos/martijnvg/elasticsearch/hooks","issue_events_url":"https://api.github.com/repos/martijnvg/elasticsearch/issues/events{/number}","events_url":"https://api.github.com/repos/martijnvg/elasticsearch/events","assignees_url":"https://api.github.com/repos/martijnvg/elasticsearch/assignees{/user}","branches_url":"https://api.github.com/repos/martijnvg/elasticsearch/branches{/branch}","tags_url":"https://api.github.com/repos/martijnvg/elasticsearch/tags","blobs_url":"https://api.github.com/repos/martijnvg/elasticsearch/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/martijnvg/elasticsearch/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/martijnvg/elasticsearch/git/refs{/sha}","trees_url":"https://api.github.com/repos/martijnvg/elasticsearch/git/trees{/sha}","statuses_url":"https://api.github.com/repos/martijnvg/elasticsearch/statuses/{sha}","languages_url":"https://api.github.com/repos/martijnvg/elasticsearch/languages","stargazers_url":"https://api.github.com/repos/martijnvg/elasticsearch/stargazers","contributors_url":"https://api.github.com/repos/martijnvg/elasticsearch/contributors","subscribers_url":"https://api.github.com/repos/martijnvg/elasticsearch/subscribers","subscription_url":"https://api.github.com/repos/martijnvg/elasticsearch/subscription","commits_url":"https://api.github.com/repos/martijnvg/elasticsearch/commits{/sha}","git_commits_url":"https://api.github.com/repos/martijnvg/elasticsearch/git/commits{/sha}","comments_url":"https://api.github.com/repos/martijnvg/elasticsearch/comments{/number}","issue_comment_url":"https://api.github.com/repos/martijnvg/elasticsearch/issues/comments/{number}","contents_url":"https://api.github.com/repos/martijnvg/elasticsearch/contents/{+path}","compare_url":"https://api.github.com/repos/martijnvg/elasticsearch/compare/{base}...{head}","merges_url":"https://api.github.com/repos/martijnvg/elasticsearch/merges","archive_url":"https://api.github.com/repos/martijnvg/elasticsearch/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/martijnvg/elasticsearch/downloads","issues_url":"https://api.github.com/repos/martijnvg/elasticsearch/issues{/number}","pulls_url":"https://api.github.com/repos/martijnvg/elasticsearch/pulls{/number}","milestones_url":"https://api.github.com/repos/martijnvg/elasticsearch/milestones{/number}","notifications_url":"https://api.github.com/repos/martijnvg/elasticsearch/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/martijnvg/elasticsearch/labels{/name}","releases_url":"https://api.github.com/repos/martijnvg/elasticsearch/releases{/id}","created_at":"2012-02-29T18:56:33Z","updated_at":"2014-08-09T22:19:31Z","pushed_at":"2015-02-16T21:21:45Z","git_url":"git://github.com/martijnvg/elasticsearch.git","ssh_url":"git@github.com:martijnvg/elasticsearch.git","clone_url":"https://github.com/martijnvg/elasticsearch.git","svn_url":"https://github.com/martijnvg/elasticsearch","homepage":"http://elasticsearch.org","size":84576,"stargazers_count":12,"watchers_count":12,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":0,"forks":2,"open_issues":0,"watchers":12,"default_branch":"master"}},"base":{"label":"elasticsearch:master","ref":"master","sha":"5b9659585414cd3da7d199550ce283a14b70c3b3","user":{"login":"elasticsearch","id":141304,"avatar_url":"https://avatars.githubusercontent.com/u/141304?v=3","gravatar_id":"","url":"https://api.github.com/users/elasticsearch","html_url":"https://github.com/elasticsearch","followers_url":"https://api.github.com/users/elasticsearch/followers","following_url":"https://api.github.com/users/elasticsearch/following{/other_user}","gists_url":"https://api.github.com/users/elasticsearch/gists{/gist_id}","starred_url":"https://api.github.com/users/elasticsearch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elasticsearch/subscriptions","organizations_url":"https://api.github.com/users/elasticsearch/orgs","repos_url":"https://api.github.com/users/elasticsearch/repos","events_url":"https://api.github.com/users/elasticsearch/events{/privacy}","received_events_url":"https://api.github.com/users/elasticsearch/received_events","type":"Organization","site_admin":false},"repo":{"id":507775,"name":"elasticsearch","full_name":"elasticsearch/elasticsearch","owner":{"login":"elasticsearch","id":141304,"avatar_url":"https://avatars.githubusercontent.com/u/141304?v=3","gravatar_id":"","url":"https://api.github.com/users/elasticsearch","html_url":"https://github.com/elasticsearch","followers_url":"https://api.github.com/users/elasticsearch/followers","following_url":"https://api.github.com/users/elasticsearch/following{/other_user}","gists_url":"https://api.github.com/users/elasticsearch/gists{/gist_id}","starred_url":"https://api.github.com/users/elasticsearch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elasticsearch/subscriptions","organizations_url":"https://api.github.com/users/elasticsearch/orgs","repos_url":"https://api.github.com/users/elasticsearch/repos","events_url":"https://api.github.com/users/elasticsearch/events{/privacy}","received_events_url":"https://api.github.com/users/elasticsearch/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/elasticsearch/elasticsearch","description":"Open Source, Distributed, RESTful Search Engine","fork":false,"url":"https://api.github.com/repos/elasticsearch/elasticsearch","forks_url":"https://api.github.com/repos/elasticsearch/elasticsearch/forks","keys_url":"https://api.github.com/repos/elasticsearch/elasticsearch/keys{/key_id}","collaborators_url":"https://api.github.com/repos/elasticsearch/elasticsearch/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/elasticsearch/elasticsearch/teams","hooks_url":"https://api.github.com/repos/elasticsearch/elasticsearch/hooks","issue_events_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/events{/number}","events_url":"https://api.github.com/repos/elasticsearch/elasticsearch/events","assignees_url":"https://api.github.com/repos/elasticsearch/elasticsearch/assignees{/user}","branches_url":"https://api.github.com/repos/elasticsearch/elasticsearch/branches{/branch}","tags_url":"https://api.github.com/repos/elasticsearch/elasticsearch/tags","blobs_url":"https://api.github.com/repos/elasticsearch/elasticsearch/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/elasticsearch/elasticsearch/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/elasticsearch/elasticsearch/git/refs{/sha}","trees_url":"https://api.github.com/repos/elasticsearch/elasticsearch/git/trees{/sha}","statuses_url":"https://api.github.com/repos/elasticsearch/elasticsearch/statuses/{sha}","languages_url":"https://api.github.com/repos/elasticsearch/elasticsearch/languages","stargazers_url":"https://api.github.com/repos/elasticsearch/elasticsearch/stargazers","contributors_url":"https://api.github.com/repos/elasticsearch/elasticsearch/contributors","subscribers_url":"https://api.github.com/repos/elasticsearch/elasticsearch/subscribers","subscription_url":"https://api.github.com/repos/elasticsearch/elasticsearch/subscription","commits_url":"https://api.github.com/repos/elasticsearch/elasticsearch/commits{/sha}","git_commits_url":"https://api.github.com/repos/elasticsearch/elasticsearch/git/commits{/sha}","comments_url":"https://api.github.com/repos/elasticsearch/elasticsearch/comments{/number}","issue_comment_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/comments/{number}","contents_url":"https://api.github.com/repos/elasticsearch/elasticsearch/contents/{+path}","compare_url":"https://api.github.com/repos/elasticsearch/elasticsearch/compare/{base}...{head}","merges_url":"https://api.github.com/repos/elasticsearch/elasticsearch/merges","archive_url":"https://api.github.com/repos/elasticsearch/elasticsearch/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/elasticsearch/elasticsearch/downloads","issues_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues{/number}","pulls_url":"https://api.github.com/repos/elasticsearch/elasticsearch/pulls{/number}","milestones_url":"https://api.github.com/repos/elasticsearch/elasticsearch/milestones{/number}","notifications_url":"https://api.github.com/repos/elasticsearch/elasticsearch/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/elasticsearch/elasticsearch/labels{/name}","releases_url":"https://api.github.com/repos/elasticsearch/elasticsearch/releases{/id}","created_at":"2010-02-08T13:20:56Z","updated_at":"2015-02-17T08:34:12Z","pushed_at":"2015-02-17T08:34:12Z","git_url":"git://github.com/elasticsearch/elasticsearch.git","ssh_url":"git@github.com:elasticsearch/elasticsearch.git","clone_url":"https://github.com/elasticsearch/elasticsearch.git","svn_url":"https://github.com/elasticsearch/elasticsearch","homepage":"http://elasticsearch.org","size":275538,"stargazers_count":9999,"watchers_count":9999,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":3086,"mirror_url":null,"open_issues_count":962,"forks":3086,"open_issues":962,"watchers":9999,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/elasticsearch/elasticsearch/pulls/9632"},"html":{"href":"https://github.com/elasticsearch/elasticsearch/pull/9632"},"issue":{"href":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9632"},"comments":{"href":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9632/comments"},"review_comments":{"href":"https://api.github.com/repos/elasticsearch/elasticsearch/pulls/9632/comments"},"review_comment":{"href":"https://api.github.com/repos/elasticsearch/elasticsearch/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/elasticsearch/elasticsearch/pulls/9632/commits"},"statuses":{"href":"https://api.github.com/repos/elasticsearch/elasticsearch/statuses/d77f74c0fde416cf613972a7bbdd84b4c0e48a88"}}}}