{"action":"created","issue":{"url":"https://api.github.com/repos/deis/deis/issues/4649","labels_url":"https://api.github.com/repos/deis/deis/issues/4649/labels{/name}","comments_url":"https://api.github.com/repos/deis/deis/issues/4649/comments","events_url":"https://api.github.com/repos/deis/deis/issues/4649/events","html_url":"https://github.com/deis/deis/issues/4649","id":112651905,"number":4649,"title":"Deis is losing a node every time.","user":{"login":"robeferre","id":6004768,"avatar_url":"https://avatars.githubusercontent.com/u/6004768?v=3","gravatar_id":"","url":"https://api.github.com/users/robeferre","html_url":"https://github.com/robeferre","followers_url":"https://api.github.com/users/robeferre/followers","following_url":"https://api.github.com/users/robeferre/following{/other_user}","gists_url":"https://api.github.com/users/robeferre/gists{/gist_id}","starred_url":"https://api.github.com/users/robeferre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robeferre/subscriptions","organizations_url":"https://api.github.com/users/robeferre/orgs","repos_url":"https://api.github.com/users/robeferre/repos","events_url":"https://api.github.com/users/robeferre/events{/privacy}","received_events_url":"https://api.github.com/users/robeferre/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/deis/deis/labels/bug","name":"bug","color":"e11d21"},{"url":"https://api.github.com/repos/deis/deis/labels/etcd/fleet","name":"etcd/fleet","color":"fef2c0"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-10-21T18:34:07Z","updated_at":"2015-11-17T17:52:36Z","closed_at":"2015-11-17T17:52:36Z","body":"Hi, we are planning to use Deis at our production environment, but I have presented some strange issues regarding my cluster nodes availability\r\n\r\nEg. This is the normal behavior of my units.\r\ncoreos001 ~ # deisctl list\r\nUNIT                            MACHINE                         LOAD    ACTIVE          SUB\r\ndeis-builder.service            c742616b.../192.168.57.66       loaded  activating      start-post\r\ndeis-controller.service         c742616b.../192.168.57.66       loaded  active          running\r\ndeis-database.service           c742616b.../192.168.57.66       loaded  active          running\r\ndeis-logger.service             c742616b.../192.168.57.66       loaded  active          running\r\ndeis-logspout.service           16a4564a.../192.168.57.65       loaded  active          running\r\ndeis-logspout.service           5ac8a25a.../192.168.57.67       loaded  active          running\r\ndeis-logspout.service           c742616b.../192.168.57.66       loaded  active          running\r\ndeis-mesos-marathon.service     c742616b.../192.168.57.66       loaded  active          running\r\ndeis-mesos-master.service       c742616b.../192.168.57.66       loaded  active          running\r\ndeis-mesos-slave.service        16a4564a.../192.168.57.65       loaded  active          running\r\ndeis-mesos-slave.service        5ac8a25a.../192.168.57.67       loaded  active          running\r\ndeis-mesos-slave.service        c742616b.../192.168.57.66       loaded  active          running\r\ndeis-publisher.service          16a4564a.../192.168.57.65       loaded  active          running\r\ndeis-publisher.service          5ac8a25a.../192.168.57.67       loaded  active          running\r\ndeis-publisher.service          c742616b.../192.168.57.66       loaded  active          running\r\ndeis-registry@1.service         c742616b.../192.168.57.66       loaded  active          running\r\ndeis-router@1.service           16a4564a.../192.168.57.65       loaded  active          running\r\ndeis-router@2.service           5ac8a25a.../192.168.57.67       loaded  active          running\r\ndeis-router@3.service           c742616b.../192.168.57.66       loaded  active          running\r\ndeis-store-daemon.service       16a4564a.../192.168.57.65       loaded  active          running\r\ndeis-store-daemon.service       5ac8a25a.../192.168.57.67       loaded  active          running\r\ndeis-store-daemon.service       c742616b.../192.168.57.66       loaded  active          running\r\ndeis-store-gateway@1.service    c742616b.../192.168.57.66       loaded  active          running\r\ndeis-store-metadata.service     16a4564a.../192.168.57.65       loaded  active          running\r\ndeis-store-metadata.service     5ac8a25a.../192.168.57.67       loaded  active          running\r\ndeis-store-metadata.service     c742616b.../192.168.57.66       loaded  active          running\r\ndeis-store-monitor.service      16a4564a.../192.168.57.65       loaded  active          running\r\ndeis-store-monitor.service      5ac8a25a.../192.168.57.67       loaded  active          running\r\ndeis-store-monitor.service      c742616b.../192.168.57.66       loaded  active          running\r\ndeis-store-volume.service       16a4564a.../192.168.57.65       loaded  active          running\r\ndeis-store-volume.service       5ac8a25a.../192.168.57.67       loaded  active          running\r\ndeis-store-volume.service       c742616b.../192.168.57.66       loaded  active          running\r\ndeis-zookeeper.service          16a4564a.../192.168.57.65       loaded  active          running\r\ndeis-zookeeper.service          5ac8a25a.../192.168.57.67       loaded  active          running\r\ndeis-zookeeper.service          c742616b.../192.168.57.66       loaded  active          running\r\n\r\nThe problem is that every time my cluster is losing a node and reconfiguring all the stuff.\r\n\r\ncoreos002 ~ # fleetctl list-units\r\nUNIT                            MACHINE                         ACTIVE          SUB\r\ndeis-builder.service            c742616b.../192.168.57.66       activating      start-post\r\ndeis-controller.service         c742616b.../192.168.57.66       active          running\r\ndeis-database.service           c742616b.../192.168.57.66       active          running\r\ndeis-logger.service             c742616b.../192.168.57.66       active          running\r\ndeis-logspout.service           5ac8a25a.../192.168.57.67       active          running\r\ndeis-logspout.service           c742616b.../192.168.57.66       active          running\r\ndeis-mesos-marathon.service     c742616b.../192.168.57.66       active          running\r\ndeis-mesos-master.service       c742616b.../192.168.57.66       activating      start-pre\r\ndeis-mesos-slave.service        5ac8a25a.../192.168.57.67       active          running\r\ndeis-mesos-slave.service        c742616b.../192.168.57.66       active          running\r\ndeis-publisher.service          5ac8a25a.../192.168.57.67       active          running\r\ndeis-publisher.service          c742616b.../192.168.57.66       active          running\r\ndeis-registry@1.service         c742616b.../192.168.57.66       active          running\r\ndeis-router@2.service           5ac8a25a.../192.168.57.67       active          running\r\ndeis-router@3.service           c742616b.../192.168.57.66       active          running\r\ndeis-store-daemon.service       5ac8a25a.../192.168.57.67       active          running\r\ndeis-store-daemon.service       c742616b.../192.168.57.66       active          running\r\ndeis-store-gateway@1.service    c742616b.../192.168.57.66       activating      start-post\r\ndeis-store-metadata.service     5ac8a25a.../192.168.57.67       active          running\r\ndeis-store-metadata.service     c742616b.../192.168.57.66       active          running\r\ndeis-store-monitor.service      5ac8a25a.../192.168.57.67       active          running\r\ndeis-store-monitor.service      c742616b.../192.168.57.66       active          running\r\ndeis-store-volume.service       5ac8a25a.../192.168.57.67       active          running\r\ndeis-store-volume.service       c742616b.../192.168.57.66       active          running\r\ndeis-zookeeper.service          5ac8a25a.../192.168.57.67       active          running\r\ndeis-zookeeper.service          c742616b.../192.168.57.66       active          running\r\nzookeeper@1.service             c742616b.../192.168.57.66       active          running\r\nzookeeper@2.service             5ac8a25a.../192.168.57.67       active          running\r\n\r\nHow could I debug this issue, seems that my fleet is losing the communication with the etcd, when I type fleetctl list-machines, the missing node is not showing.\r\n\r\nThis issue is a little painfull since we are planning to use Deis at our production environment. Is there any way to make the nodes tottally available?\r\n\r\nAfter 5 to 10 mins, the node is recovered, but I think its not a common behaviour."},"comment":{"url":"https://api.github.com/repos/deis/deis/issues/comments/157451711","html_url":"https://github.com/deis/deis/issues/4649#issuecomment-157451711","issue_url":"https://api.github.com/repos/deis/deis/issues/4649","id":157451711,"user":{"login":"robeferre","id":6004768,"avatar_url":"https://avatars.githubusercontent.com/u/6004768?v=3","gravatar_id":"","url":"https://api.github.com/users/robeferre","html_url":"https://github.com/robeferre","followers_url":"https://api.github.com/users/robeferre/followers","following_url":"https://api.github.com/users/robeferre/following{/other_user}","gists_url":"https://api.github.com/users/robeferre/gists{/gist_id}","starred_url":"https://api.github.com/users/robeferre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robeferre/subscriptions","organizations_url":"https://api.github.com/users/robeferre/orgs","repos_url":"https://api.github.com/users/robeferre/repos","events_url":"https://api.github.com/users/robeferre/events{/privacy}","received_events_url":"https://api.github.com/users/robeferre/received_events","type":"User","site_admin":false},"created_at":"2015-11-17T17:52:36Z","updated_at":"2015-11-17T17:52:36Z","body":"I was not using enought ram to run the cluster, as soon as I increased the ram size this issue was solved."}}