{"action":"opened","issue":{"url":"https://api.github.com/repos/EventStore/EventStore/issues/686","labels_url":"https://api.github.com/repos/EventStore/EventStore/issues/686/labels{/name}","comments_url":"https://api.github.com/repos/EventStore/EventStore/issues/686/comments","events_url":"https://api.github.com/repos/EventStore/EventStore/issues/686/events","html_url":"https://github.com/EventStore/EventStore/issues/686","id":106943724,"number":686,"title":"Moving from single cluster node to multiple causing AccessDenied on $projections-$master","user":{"login":"ianbattersby","id":185776,"avatar_url":"https://avatars.githubusercontent.com/u/185776?v=3","gravatar_id":"","url":"https://api.github.com/users/ianbattersby","html_url":"https://github.com/ianbattersby","followers_url":"https://api.github.com/users/ianbattersby/followers","following_url":"https://api.github.com/users/ianbattersby/following{/other_user}","gists_url":"https://api.github.com/users/ianbattersby/gists{/gist_id}","starred_url":"https://api.github.com/users/ianbattersby/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ianbattersby/subscriptions","organizations_url":"https://api.github.com/users/ianbattersby/orgs","repos_url":"https://api.github.com/users/ianbattersby/repos","events_url":"https://api.github.com/users/ianbattersby/events{/privacy}","received_events_url":"https://api.github.com/users/ianbattersby/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-09-17T09:19:11Z","updated_at":"2015-09-17T09:19:11Z","closed_at":null,"body":"This may have been a stupid thing to do but I changed a single cluster-node setup to 3-node cluster setup. The data seems to have propagated to other nodes successfully but I see this on start-up:\r\n\r\n```\r\n[09048,17,09:12:36.585] There are FEW MASTERS according to gossip, need to start elections. MASTER: [InstanceId: {168189dc-d6ac-4f\r\n40-8314-2e112cb9c230}, InternalTcp: 127.0.0.1:2111, InternalSecureTcp: , ExternalTcp: 127.0.0.1:2113, ExternalSecureTcp: , Interna\r\nlHttp: 127.0.0.1:2112, ExternalHttp: 127.0.0.1:2114].\r\nGOSSIP:\r\nVND {070a878f-d76d-4d88-b0e0-49a5c6ae5203} <DEAD> [PreReplica, 127.0.0.1:4111, n/a, 10.50.1.71:4113, n/a, 127.0.0.1:4112, 10.50.1.\r\n71:4114] 19174/19665/19665/E0@0:{0e6b5917-ff2a-4e1e-9259-b077b220f50f} | 2015-09-17 09:12:36.588\r\n[27016,18,09:12:36.562] ========== [127.0.0.1:3112] IS SHUTTING DOWN...VND {c2088777-1f8f-4523-b04f-5f60ec3cf9f3} <LIVE> [Master,\r\n127.0.0.1:3111, n/a, 127.0.0.1:3113, n/a, 127.0.0.1:3112, 127.0.0.1:3114] 19174/19665/19665/E0@0:{0e6b5917-ff2a-4e1e-9259-b077b220\r\nf50f} | 2015-09-17 09:12:36.530\r\n\r\n[27016,20,09:12:36.594] Cannot write commands to the stream $projections-$master. status: AccessDeniedVND {168189dc-d6ac-4f40-8314\r\n-2e112cb9c230} <LIVE> [Master, 127.0.0.1:2111, n/a, 127.0.0.1:2113, n/a, 127.0.0.1:2112, 127.0.0.1:2114] 63466567/63466955/6346672\r\n6/E9@63465886:{36bc5bab-79d6-4790-a9d6-468d5ea2aaf3} | 2015-09-17 09:12:36.401\r\n\r\n[27016,22,09:12:36.594] Cannot write commands to the stream $projections-$master. status: AccessDenied[09048,30,09:12:36.618] Conn\r\nection 'internal-normal' [127.0.0.1:47061, {2edcb019-7e32-49cd-89fd-2580813faf70}] closed: Success.\r\n\r\n[27016,22,09:12:36.594] Error while processing message WRITE COMPLETED: CorrelationId: 0c5fc1b1-5723-4172-b320-77b2712673a1, Resul\r\nt: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projectio\r\nn Core #2'.[09048,11,09:12:36.618] Lost connection from 127.0.0.1:47061\r\n\r\n[09048,30,09:12:36.625] ES TcpConnection closed [09:12:36.625: N127.0.0.1:47060, L127.0.0.1:2111, {4c9ffb2b-99ce-4dd8-90f3-dbf92ea\r\nad386}]:Cannot write commands to the stream $projections-$master. status: AccessDenied\r\n\r\nReceived bytes: 630, Sent bytes: 3975599\r\n[27016,20,09:12:36.594] Error while processing message WRITE COMPLETED: CorrelationId: e3cc96b0-c9f6-44fa-8d77-160f8ffb105c, Resul\r\nt: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Projectio\r\nn Core #0'.Send calls: 958, callbacks: 957\r\nReceive calls: 5, callbacks: 5\r\n\r\nCannot write commands to the stream $projections-$master. status: AccessDeniedClose reason: [Success] Socket closed\r\n\r\n[27016,21,09:12:36.619] Cannot write commands to the stream $projections-$master. status: AccessDenied\r\n\r\n[27016,21,09:12:36.619] Error while processing message WRITE COMPLETED: CorrelationId: 39bbbe88-47c1-4d0c-a898-d57e0c1d2507, Resul\r\nt: AccessDenied, Message: Access denied., FirstEventNumber: -2147483648, LastEventNumber: -2147483648 in queued handler 'Proje[090\r\n48,30,09:12:36.625] Connection 'internal-normal' [127.0.0.1:47060, {4c9ffb2b-99ce-4dd8-90f3-dbf92eaad386}] closed: Success.ction C\r\nore #1'.\r\n[09048,17,09:12:36.625] Looks like node [127.0.0.1:3111] is DEAD (TCP connection lost).\r\n```\r\n\r\nAlso, I end up in this endless repeating loop thereafter:\r\n```\r\n--------------------------------------------------------------------------------[28388,32,09:14:06.696] Connection 'internal-norma\r\nl' [127.0.0.1:47164, {4d3b94b3-344e-4ffd-88a8-01997a0e8af6}] closed: ConnectionAborted.[14024,17,09:14:06.732] ========== [127.0.0\r\n.1:4112] Service 'Storage Chaser' has shut down.\r\n\r\n[14024,17,09:14:06.732] ========== [127.0.0.1:4112] All Services Shutdown.\r\n[28388,06,09:14:06.696] Lost connection from 127.0.0.1:47165\r\n\r\n[28388,06,09:14:06.696] Lost connection from 127.0.0.1:47164\r\n[14024,17,09:14:06.738] ========== [127.0.0.1:4112] IS SHUT DOWN.\r\n[14024,09,09:14:06.738] Subscriptions received state change to Shutdown stopping listening.\r\n[14024,17,09:14:06.753] Exiting with exit code: 0.\r\nExit reason: Shutdown and exit from process was requested.\r\n[28660,18,09:14:06.722] ========== [127.0.0.1:3112] Service 'Storage Chaser' has shut down.\r\n[28660,18,09:14:06.722] ========== [127.0.0.1:3112] All Services Shutdown.\r\n[28660,18,09:14:06.737] ========== [127.0.0.1:3112] IS SHUT DOWN.\r\n[28660,07,09:14:06.737] Subscriptions received state change to Shutdown stopping listening.\r\n[28660,18,09:14:06.761] Exiting with exit code: 0.\r\nExit reason: Shutdown and exit from process was requested.\r\n[28388,17,09:14:07.396] ELECTIONS: (V=3) TIMED OUT! (S=ElectingLeader, M=).\r\n[28388,17,09:14:07.396] ELECTIONS: (V=4) SHIFT TO LEADER ELECTION.\r\n[28388,17,09:14:07.396] ELECTIONS: (V=4) VIEWCHANGE FROM [127.0.0.1:2112, {e4dd2106-3100-47ea-b9bf-ae3b486400eb}].\r\n[28388,17,09:14:08.397] ELECTIONS: (V=4) TIMED OUT! (S=ElectingLeader, M=).\r\n[28388,17,09:14:08.397] ELECTIONS: (V=5) SHIFT TO LEADER ELECTION.\r\n[28388,17,09:14:08.397] ELECTIONS: (V=5) VIEWCHANGE FROM [127.0.0.1:2112, {e4dd2106-3100-47ea-b9bf-ae3b486400eb}].\r\n[28388,17,09:14:08.645] IDEMPOTENT WRITE TO STREAM ClientCorrelationID 0d9b8286-5d32-45d8-b541-2a49965fbaa0, EventStreamId: $proje\r\nctions-$master, CorrelationId: 0d9b8286-5d32-45d8-b541-2a49965fbaa0, FirstEventNumber: 15536, LastEventNumber: 15536.\r\n[28388,17,09:14:08.645] IDEMPOTENT WRITE TO STREAM ClientCorrelationID 0a7ebc21-d952-4c8b-92e1-502a7ff01449, EventStreamId: $proje\r\nctions-$master, CorrelationId: 0a7ebc21-d952-4c8b-92e1-502a7ff01449, FirstEventNumber: 15537, LastEventNumber: 15537.\r\n[28388,17,09:14:08.645] IDEMPOTENT WRITE TO STREAM ClientCorrelationID e36b2432-aa56-448c-b8f4-32034b730247, EventStreamId: $proje\r\nctions-$master, CorrelationId: e36b2432-aa56-448c-b8f4-32034b730247, FirstEventNumber: 15535, LastEventNumber: 15535.\r\n[28388,17,09:14:09.398] ELECTIONS: (V=5) TIMED OUT! (S=ElectingLeader, M=).\r\n[28388,17,09:14:09.398] ELECTIONS: (V=6) SHIFT TO LEADER ELECTION.\r\n[28388,17,09:14:09.398] ELECTIONS: (V=6) VIEWCHANGE FROM [127.0.0.1:2112, {e4dd2106-3100-47ea-b9bf-ae3b486400eb}].\r\n[28388,17,09:14:09.686] === NO QUORUM EMERGED WITHIN TIMEOUT... RETIRING...\r\n[28388,17,09:14:09.686] ========== [127.0.0.1:2112] IS UNKNOWN...\r\n[28388,09,09:14:09.686] Subscriptions received state change to Unknown stopping listening.\r\n[28388,19,09:14:09.701] _eventReaders is not empty after all the projections have been killed\r\n[28388,20,09:14:09.701] _eventReaders is not empty after all the projections have been killed\r\n[28388,21,09:14:09.701] _eventReaders is not empty after all the projections have been killed\r\n[28388,17,09:14:10.399] ELECTIONS: (V=6) TIMED OUT! (S=ElectingLeader, M=).\r\n[28388,17,09:14:10.399] ELECTIONS: (V=7) SHIFT TO LEADER ELECTION.\r\n[28388,17,09:14:10.399] ELECTIONS: (V=7) VIEWCHANGE FROM [127.0.0.1:2112, {e4dd2106-3100-47ea-b9bf-ae3b486400eb}].\r\n[28388,17,09:14:11.400] ELECTIONS: (V=7) TIMED OUT! (S=ElectingLeader, M=).\r\n[28388,17,09:14:11.400] ELECTIONS: (V=8) SHIFT TO LEADER ELECTION.\r\n[28388,17,09:14:11.400] ELECTIONS: (V=8) VIEWCHANGE FROM [127.0.0.1:2112, {e4dd2106-3100-47ea-b9bf-ae3b486400eb}].\r\n[28388,17,09:14:12.401] ELECTIONS: (V=8) TIMED OUT! (S=ElectingLeader, M=).\r\n[28388,17,09:14:12.401] ELECTIONS: (V=9) SHIFT TO LEADER ELECTION.\r\n[28388,17,09:14:12.401] ELECTIONS: (V=9) VIEWCHANGE FROM [127.0.0.1:2112, {e4dd2106-3100-47ea-b9bf-ae3b486400eb}].\r\n<snip>\r\n[28388,17,09:14:32.410] ELECTIONS: (V=29) SHIFT TO LEADER ELECTION.\r\n[28388,17,09:14:32.410] ELECTIONS: (V=29) VIEWCHANGE FROM [127.0.0.1:2112, {e4dd2106-3100-47ea-b9bf-ae3b486400eb}].\r\n[28388,17,09:14:33.411] ELECTIONS: (V=29) TIMED OUT! (S=ElectingLeader, M=).\r\n```\r\n\r\nI don't know if this is expected behaviour and I've just not followed the correct route?"}}