{"action":"created","issue":{"url":"https://api.github.com/repos/EventStore/EventStore/issues/416","labels_url":"https://api.github.com/repos/EventStore/EventStore/issues/416/labels{/name}","comments_url":"https://api.github.com/repos/EventStore/EventStore/issues/416/comments","events_url":"https://api.github.com/repos/EventStore/EventStore/issues/416/events","html_url":"https://github.com/EventStore/EventStore/issues/416","id":60283053,"number":416,"title":"DNS cluster config doesn't say what's wrong","user":{"login":"haf","id":193115,"avatar_url":"https://avatars.githubusercontent.com/u/193115?v=3","gravatar_id":"","url":"https://api.github.com/users/haf","html_url":"https://github.com/haf","followers_url":"https://api.github.com/users/haf/followers","following_url":"https://api.github.com/users/haf/following{/other_user}","gists_url":"https://api.github.com/users/haf/gists{/gist_id}","starred_url":"https://api.github.com/users/haf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/haf/subscriptions","organizations_url":"https://api.github.com/users/haf/orgs","repos_url":"https://api.github.com/users/haf/repos","events_url":"https://api.github.com/users/haf/events{/privacy}","received_events_url":"https://api.github.com/users/haf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-03-08T23:38:47Z","updated_at":"2015-03-08T23:46:52Z","closed_at":null,"body":"I'm trying to get the cluster up and running, but it doesn't seem to want to play. This is what it looks like directly when starting ES:\r\n\r\n![screen shot 2015-03-09 at 00 32 23](https://cloud.githubusercontent.com/assets/193115/6548307/6499569c-c5f4-11e4-88b1-8fae603f9442.png)\r\n\r\nAll configuration in text from a node:\r\n\r\n```\r\nStarting on internal ep: 10.1.0.29:{1112,2112}, external ep: 10.1.0.29:{1113,2113}, with GOSSIP_SEED=10.1.0.29:2112\r\n2015-03-08 23:35:01.8811|INFO|ProgramBase`1|\r\nES VERSION:               3.0.1.0 (release-oss-v3.0.1/7fa876c111888dd5980dbd86d126e6abe13b05ab, Thu, 23 Oct 2014 22:27:04 +0100)\r\nOS:                       Unknown (Unix 3.18.1.0)\r\nRUNTIME:                  3.8.0 ((detached/45d0ba1) (64-bit)\r\nGC:                       2 GENERATIONS\r\nLOGS:                     /var/lib/eventstore-logs\r\n\r\nHELP:                     False (<DEFAULT>)\r\nVERSION:                  False (<DEFAULT>)\r\nLOG:                      <empty> (<DEFAULT>)\r\nCONFIG:                   <empty> (<DEFAULT>)\r\nDEFINES:                  <empty> (<DEFAULT>)\r\nWHAT IF:                  False (<DEFAULT>)\r\nINT IP:                   10.1.0.29 (Command Line)\r\nEXT IP:                   10.1.0.29 (Command Line)\r\nINT HTTP PORT:            2112 (Command Line)\r\nEXT HTTP PORT:            2113 (Command Line)\r\nINT TCP PORT:             1112 (Command Line)\r\nINT SECURE TCP PORT:      0 (<DEFAULT>)\r\nEXT TCP PORT:             1113 (Command Line)\r\nEXT SECURE TCP PORT:      0 (<DEFAULT>)\r\nINT TCP HEARTBEAT TIMEOUT: 700 (<DEFAULT>)\r\nEXT TCP HEARTBEAT TIMEOUT: 1000 (<DEFAULT>)\r\nINT TCP HEARTBEAT INTERVAL: 700 (<DEFAULT>)\r\nEXT TCP HEARTBEAT INTERVAL: 2000 (<DEFAULT>)\r\nFORCE:                    False (<DEFAULT>)\r\nCLUSTER SIZE:             3 (Command Line)\r\nNODE PRIORITY:            0 (<DEFAULT>)\r\nMIN FLUSH DELAY MS:       2 (<DEFAULT>)\r\nCOMMIT COUNT:             -1 (<DEFAULT>)\r\nPREPARE COUNT:            -1 (<DEFAULT>)\r\nADMIN ON EXT:             True (<DEFAULT>)\r\nSTATS ON EXT:             True (<DEFAULT>)\r\nGOSSIP ON EXT:            True (<DEFAULT>)\r\nDISABLE SCAVENGE MERGING: False (<DEFAULT>)\r\nDISCOVER VIA DNS:         True (<DEFAULT>)\r\nCLUSTER DNS:              eventstore.prod.deis.local (Command Line)\r\nCLUSTER GOSSIP PORT:      2112 (Command Line)\r\nGOSSIP SEED:              <empty> (<DEFAULT>)\r\nSTATS PERIOD SEC:         30 (<DEFAULT>)\r\nCACHED CHUNKS:            -1 (<DEFAULT>)\r\nCHUNKS CACHE SIZE:        536871424 (<DEFAULT>)\r\nMAX MEM TABLE SIZE:       1000000 (<DEFAULT>)\r\nDB:                       /var/lib/eventstore (Command Line)\r\nMEM DB:                   False (<DEFAULT>)\r\nSKIP DB VERIFY:           False (<DEFAULT>)\r\nRUN PROJECTIONS:          All (Command Line)\r\nPROJECTION THREADS:       3 (<DEFAULT>)\r\nWORKER THREADS:           5 (<DEFAULT>)\r\nHTTP PREFIXES:            <empty> (<DEFAULT>)\r\nENABLE TRUSTED AUTH:      False (<DEFAULT>)\r\nCERTIFICATE STORE LOCATION: <empty> (<DEFAULT>)\r\nCERTIFICATE STORE NAME:   <empty> (<DEFAULT>)\r\nCERTIFICATE SUBJECT NAME: <empty> (<DEFAULT>)\r\nCERTIFICATE THUMBPRINT:   <empty> (<DEFAULT>)\r\nCERTIFICATE FILE:         <empty> (<DEFAULT>)\r\nCERTIFICATE PASSWORD:     <empty> (<DEFAULT>)\r\nUSE INTERNAL SSL:         False (<DEFAULT>)\r\nSSL TARGET HOST:          n/a (<DEFAULT>)\r\nSSL VALIDATE SERVER:      True (<DEFAULT>)\r\nAUTHENTICATION TYPE:      internal (<DEFAULT>)\r\nPREPARE TIMEOUT MS:       2000 (<DEFAULT>)\r\nCOMMIT TIMEOUT MS:        2000 (<DEFAULT>)\r\nUNSAFE DISABLE FLUSH TO DISK: False (<DEFAULT>)\r\nGOSSIP INTERVAL MS:       1000 (<DEFAULT>)\r\nGOSSIP ALLOWED DIFFERENCE MS: 60000 (<DEFAULT>)\r\nGOSSIP TIMEOUT MS:        500 (<DEFAULT>)\r\n\r\n2015-03-08 23:35:01.9085|INFO|ProgramBase`1|Quorum size set to 2\r\n2015-03-08 23:35:01.9132|INFO|ProgramBase`1|Can't find plugins path: plugins\r\n2015-03-08 23:35:01.9315|INFO|ProgramBase`1|\r\nINSTANCE ID:              35e32dfa-0313-4923-9f99-26c9f5d194c9\r\nDATABASE:                 /var/lib/eventstore\r\nWRITER CHECKPOINT:        0 (0x0)\r\nCHASER CHECKPOINT:        0 (0x0)\r\nEPOCH CHECKPOINT:         -1 (0xFFFFFFFFFFFFFFFF)\r\nTRUNCATE CHECKPOINT:      -1 (0xFFFFFFFFFFFFFFFF)\r\n\r\n2015-03-08 23:35:02.1895|INFO|MiniWeb|Starting MiniWeb for /web/es/js/projections ==> /opt/eventstore/projections\r\n2015-03-08 23:35:02.1901|INFO|MiniWeb|Starting MiniWeb for /web/es/js/projections/v8/Prelude ==> /opt/eventstore/Prelude\r\n2015-03-08 23:35:02.1901|INFO|MiniWeb|Starting MiniWeb for /web/es/js/projections/resources ==> /opt/eventstore/web-resources/js\r\n2015-03-08 23:35:02.2037|INFO|MiniWeb|Starting MiniWeb for /web ==> /opt/eventstore/clusternode-web\r\n2015-03-08 23:35:02.2046|INFO|MiniWeb|Starting MiniWeb for /web/users ==> /opt/eventstore/Users/web\r\n2015-03-08 23:35:02.2046|INFO|MiniWeb|Starting MiniWeb for /web ==> /opt/eventstore/clusternode-web\r\n2015-03-08 23:35:02.2046|INFO|MiniWeb|Starting MiniWeb for /web/users ==> /opt/eventstore/Users/web\r\n2015-03-08 23:35:02.2066|INFO|ClusterVNodeController|========== [10.1.0.29:2112] SYSTEM INIT...\r\n2015-03-08 23:35:02.2163|INFO|TcpServerListener|Starting Normal TCP listening on TCP endpoint: 10.1.0.29:1113.\r\n2015-03-08 23:35:02.2196|INFO|IndexCommitter|TableIndex initialization...\r\n2015-03-08 23:35:02.2231|INFO|IndexCommitter|ReadIndex building...\r\n2015-03-08 23:35:02.2315|INFO|TcpServerListener|Starting Normal TCP listening on TCP endpoint: 10.1.0.29:1112.\r\n2015-03-08 23:35:02.2328|INFO|HttpAsyncServer|Starting HTTP server on [http://10.1.0.29:2113/]...\r\n2015-03-08 23:35:02.2352|INFO|HttpAsyncServer|HTTP server is up and listening on [http://10.1.0.29:2113/]\r\n2015-03-08 23:35:02.2352|INFO|HttpAsyncServer|Starting HTTP server on [http://10.1.0.29:2112/]...\r\n2015-03-08 23:35:02.2352|INFO|HttpAsyncServer|HTTP server is up and listening on [http://10.1.0.29:2112/]\r\n2015-03-08 23:35:02.2376|INFO|ClusterVNodeController|========== [10.1.0.29:2112] Service 'StorageReader' initialized.\r\n2015-03-08 23:35:02.2388|INFO|ClusterVNodeController|========== [10.1.0.29:2112] Service 'StorageWriter' initialized.\r\n2015-03-08 23:35:02.2396|INFO|ClusterVNodeController|========== [10.1.0.29:2112] Service 'StorageChaser' initialized.\r\n2015-03-08 23:35:02.2406|INFO|ClusterVNodeController|========== [10.1.0.29:2112] SYSTEM START...\r\n2015-03-08 23:35:02.2435|INFO|ClusterVNodeController|========== [10.1.0.29:2112] IS UNKNOWN!!! WHOA!!!\r\n```\r\n\r\nOne one node:\r\n\r\n```\r\n$ sudo netstat -napt | grep 1112\r\ntcp        0      0 172.17.8.100:1112       0.0.0.0:*               LISTEN      22429/docker-proxy\r\n```\r\n\r\nSpawning another container and trying to access the configured IP from there, to verify there's no connectivity issue:\r\n\r\n![screen shot 2015-03-09 at 00 37 19](https://cloud.githubusercontent.com/assets/193115/6548315/8098aa46-c5f4-11e4-89c8-62cfdaa74c6c.png)\r\n![screen shot 2015-03-09 at 00 34 39](https://cloud.githubusercontent.com/assets/193115/6548318/845b3194-c5f4-11e4-8dea-4591714f1254.png)\r\n\r\nVerifying DNS recods:\r\n\r\n```\r\nroot@99101d09675d:/# dig +short A eventstore.prod.deis.local\r\n172.17.8.101\r\n172.17.8.100\r\n172.17.8.102\r\n```\r\n\r\nRepro [here](https://github.com/haf/dockerfiles/tree/master/eventstore)."},"comment":{"url":"https://api.github.com/repos/EventStore/EventStore/issues/comments/77782701","html_url":"https://github.com/EventStore/EventStore/issues/416#issuecomment-77782701","issue_url":"https://api.github.com/repos/EventStore/EventStore/issues/416","id":77782701,"user":{"login":"haf","id":193115,"avatar_url":"https://avatars.githubusercontent.com/u/193115?v=3","gravatar_id":"","url":"https://api.github.com/users/haf","html_url":"https://github.com/haf","followers_url":"https://api.github.com/users/haf/followers","following_url":"https://api.github.com/users/haf/following{/other_user}","gists_url":"https://api.github.com/users/haf/gists{/gist_id}","starred_url":"https://api.github.com/users/haf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/haf/subscriptions","organizations_url":"https://api.github.com/users/haf/orgs","repos_url":"https://api.github.com/users/haf/repos","events_url":"https://api.github.com/users/haf/events{/privacy}","received_events_url":"https://api.github.com/users/haf/received_events","type":"User","site_admin":false},"created_at":"2015-03-08T23:46:52Z","updated_at":"2015-03-08T23:46:52Z","body":"It seems that all respond:\r\n\r\n```\r\nroot@99101d09675d:/# telnet 172.17.8.100 1113\r\nTrying 172.17.8.100...\r\nConnected to 172.17.8.100.\r\nEscape character is '^]'.\r\n^CConnection closed by foreign host.\r\nroot@99101d09675d:/# telnet 172.17.8.100 1112\r\nTrying 172.17.8.100...\r\nConnected to 172.17.8.100.\r\nEscape character is '^]'.\r\n^CConnection closed by foreign host.\r\nroot@99101d09675d:/# telnet 172.17.8.100 2112\r\nTrying 172.17.8.100...\r\nConnected to 172.17.8.100.\r\nEscape character is '^]'.\r\n^C^[[A^\\^]\r\ntelnet> q\r\nConnection closed.\r\nroot@99101d09675d:/# telnet 172.17.8.100 2113\r\nTrying 172.17.8.100...\r\nConnected to 172.17.8.100.\r\nEscape character is '^]'.\r\n^]\r\n```"}}