{"action":"created","issue":{"url":"https://api.github.com/repos/hopshadoop/hops/issues/99","labels_url":"https://api.github.com/repos/hopshadoop/hops/issues/99/labels{/name}","comments_url":"https://api.github.com/repos/hopshadoop/hops/issues/99/comments","events_url":"https://api.github.com/repos/hopshadoop/hops/issues/99/events","html_url":"https://github.com/hopshadoop/hops/issues/99","id":122506832,"number":99,"title":"the leader does not guaranty that there won't be two leaders at the same time","user":{"login":"berthoug","id":9653655,"avatar_url":"https://avatars.githubusercontent.com/u/9653655?v=3","gravatar_id":"","url":"https://api.github.com/users/berthoug","html_url":"https://github.com/berthoug","followers_url":"https://api.github.com/users/berthoug/followers","following_url":"https://api.github.com/users/berthoug/following{/other_user}","gists_url":"https://api.github.com/users/berthoug/gists{/gist_id}","starred_url":"https://api.github.com/users/berthoug/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/berthoug/subscriptions","organizations_url":"https://api.github.com/users/berthoug/orgs","repos_url":"https://api.github.com/users/berthoug/repos","events_url":"https://api.github.com/users/berthoug/events{/privacy}","received_events_url":"https://api.github.com/users/berthoug/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/hopshadoop/hops/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"smkniazi","id":4435820,"avatar_url":"https://avatars.githubusercontent.com/u/4435820?v=3","gravatar_id":"","url":"https://api.github.com/users/smkniazi","html_url":"https://github.com/smkniazi","followers_url":"https://api.github.com/users/smkniazi/followers","following_url":"https://api.github.com/users/smkniazi/following{/other_user}","gists_url":"https://api.github.com/users/smkniazi/gists{/gist_id}","starred_url":"https://api.github.com/users/smkniazi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smkniazi/subscriptions","organizations_url":"https://api.github.com/users/smkniazi/orgs","repos_url":"https://api.github.com/users/smkniazi/repos","events_url":"https://api.github.com/users/smkniazi/events{/privacy}","received_events_url":"https://api.github.com/users/smkniazi/received_events","type":"User","site_admin":false},"milestone":null,"comments":2,"created_at":"2015-12-16T13:49:17Z","updated_at":"2015-12-17T10:32:23Z","closed_at":null,"body":"Some logs suggest that a node can loose its leader status while it manage to update its counter less than 2*hbPeriod before the lost:\r\n\r\n**Node1:**\r\n>15/12/16 06:49:42 DEBUG [Thread-8] (LeaderElection.java:run(76))   - LE Status: id 1 Leader Election Start Round. Time period 1000 Time since last update 1001\r\n15/12/16 06:49:42 DEBUG [Thread-8] (LeaderElection.java:run(132))   - LE Status: id 1 Leader Election End Round. Update time was 6 ms\r\n15/12/16 06:49:43 DEBUG [Thread-8] (LeaderElection.java:run(76))   - LE Status: id 1 Leader Election Start Round. Time period 1000 Time since last update 1002\r\n15/12/16 06:49:45 ERROR [Thread-8] (LeaderElection.java:run(126))   - LE Status: id 1 LeaderElection: Update Tx took very long time to update: 1646, time_perid is 1000\r\n15/12/16 06:49:45 DEBUG [Thread-8] (LeaderElection.java:run(132))   - LE Status: id 1 Leader Election End Round. Update time was 1646 ms\r\n15/12/16 06:49:45 DEBUG [Thread-8] (LeaderElection.java:run(76))   - LE Status: id 1 Leader Election Start Round. Time period 1000 Time since last update 7\r\n**15/12/16 06:49:45 DEBUG [Thread-8] (LeaderElection.java:run(132))   - LE Status: id 1 Leader Election End Round. Update time was 10 ms\r\n15/12/16 06:49:46 DEBUG [Thread-8] (LeaderElection.java:run(76))   - LE Status: id 1 Leader Election Start Round. Time period 1000 Time since last update 1005\r\n15/12/16 06:49:46  INFO [Thread-8] (LETransaction.java:leaderCheck(257))  LEADER_ELECTION - LE Status: id 3 I am a NON_LEADER process \r\n15/12/16 06:49:46 DEBUG [Thread-8] (LeaderElection.java:run(132))   - LE Status: id 3 Leader Election End Round. Update time was 6 ms**\r\n15/12/16 06:49:47 DEBUG [Thread-8] (LeaderElection.java:run(76))   - LE Status: id 3 Leader Election Start Round. Time period 1000 Time since last update 1002\r\n15/12/16 06:49:47 DEBUG [Thread-8] (LeaderElection.java:run(132))   - LE Status: id 3 Leader Election End Round. Update time was 7 ms\r\n15/12/16 06:49:48 DEBUG [Thread-8] (LeaderElection.java:run(76))   - LE Status: id 3 Leader Election Start Round. Time period 1000 Time since last update 1002\r\n15/12/16 06:49:48 DEBUG [Thread-8] (LeaderElection.java:run(132))   - LE Status: id 3 Leader Election End Round. Update time was 7 ms\r\n\r\n**Node2:**\r\n>15/12/16 06:49:46 DEBUG [Thread-8] (LeaderElection.java:run(76))   - LE Status: id 2 Leader Election Start Round. Time period 1000 Time since last update 1002\r\n15/12/16 06:49:46 DEBUG [Thread-8] (LeaderElection.java:run(132))   - LE Status: id 2 Leader Election End Round. Update time was 7 ms\r\n**15/12/16 06:49:47 DEBUG [Thread-8] (LeaderElection.java:run(76))   - LE Status: id 2 Leader Election Start Round. Time period 1000 Time since last update 1003\r\n15/12/16 06:49:47  WARN [Thread-8] (LETransaction.java:leaderCheck(250))  LEADER_ELECTION - LE Status: id 2 I can be the leader but I have weak locks. Retry with stronger lock\r\n15/12/16 06:49:47  WARN [Thread-8] (LETransaction.java:periodicUpdate(156))  LEADER_ELECTION - LE Status: id 2 periodic update. Stronger locks requested in next round\r\n15/12/16 06:49:47 DEBUG [Thread-8] (LeaderElection.java:run(132))   - LE Status: id 2 Leader Election End Round. Update time was 8 ms\r\n15/12/16 06:49:47 DEBUG [Thread-8] (LeaderElection.java:run(76))   - LE Status: id 2 Leader Election Start Round. Time period 1000 Time since last update 2\r\n15/12/16 06:49:47  INFO [Thread-8] (LETransaction.java:leaderCheck(243))  LEADER_ELECTION - LE Status: id 2 I am the new LEADER. \r\n15/12/16 06:49:47 DEBUG [Thread-8] (LETransaction.java:removeDeadNameNodes(290))  LEADER_ELECTION - LE Status: id 2 removing dead node 1\r\n15/12/16 06:49:47 DEBUG [Thread-8] (LeaderElection.java:run(132))   - LE Status: id 2 Leader Election End Round. Update time was 7 ms**\r\n15/12/16 06:49:48 DEBUG [Thread-8] (LeaderElection.java:run(76))   - LE Status: id 2 Leader Election Start Round. Time period 1000 Time since last update 1002\r\n15/12/16 06:49:48 DEBUG [Thread-8] (LeaderElection.java:run(132))   - LE Status: id 2 Leader Election End Round. Update time was 12 ms\r\n15/12/16 06:49:49 DEBUG [Thread-8] (LeaderElection.java:run(76))   - LE Status: id 2 Leader Election Start Round. Time period 1000 Time since last update 1003\r\n15/12/16 06:49:49 DEBUG [Thread-8] (LeaderElection.java:run(132))   - LE Status: id 2 Leader Election End Round. Update time was 27 ms\r\n\r\n\r\n(The nodes clock may not be exactly in sync)\r\n"},"comment":{"url":"https://api.github.com/repos/hopshadoop/hops/issues/comments/165413085","html_url":"https://github.com/hopshadoop/hops/issues/99#issuecomment-165413085","issue_url":"https://api.github.com/repos/hopshadoop/hops/issues/99","id":165413085,"user":{"login":"jimdowling","id":1904928,"avatar_url":"https://avatars.githubusercontent.com/u/1904928?v=3","gravatar_id":"","url":"https://api.github.com/users/jimdowling","html_url":"https://github.com/jimdowling","followers_url":"https://api.github.com/users/jimdowling/followers","following_url":"https://api.github.com/users/jimdowling/following{/other_user}","gists_url":"https://api.github.com/users/jimdowling/gists{/gist_id}","starred_url":"https://api.github.com/users/jimdowling/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jimdowling/subscriptions","organizations_url":"https://api.github.com/users/jimdowling/orgs","repos_url":"https://api.github.com/users/jimdowling/repos","events_url":"https://api.github.com/users/jimdowling/events{/privacy}","received_events_url":"https://api.github.com/users/jimdowling/received_events","type":"User","site_admin":false},"created_at":"2015-12-17T10:32:23Z","updated_at":"2015-12-17T10:32:23Z","body":"What does this mean? Why would the RM volutnarily relinquish its role as leader?"}}