{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/2592","labels_url":"https://api.github.com/repos/coreos/etcd/issues/2592/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/2592/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/2592/events","html_url":"https://github.com/coreos/etcd/issues/2592","id":64846314,"number":2592,"title":"Running back-2-back tests split-brain condition, unrecoverable. ","user":{"login":"timothysc","id":169553,"avatar_url":"https://avatars.githubusercontent.com/u/169553?v=3","gravatar_id":"","url":"https://api.github.com/users/timothysc","html_url":"https://github.com/timothysc","followers_url":"https://api.github.com/users/timothysc/followers","following_url":"https://api.github.com/users/timothysc/following{/other_user}","gists_url":"https://api.github.com/users/timothysc/gists{/gist_id}","starred_url":"https://api.github.com/users/timothysc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timothysc/subscriptions","organizations_url":"https://api.github.com/users/timothysc/orgs","repos_url":"https://api.github.com/users/timothysc/repos","events_url":"https://api.github.com/users/timothysc/events{/privacy}","received_events_url":"https://api.github.com/users/timothysc/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-03-27T20:36:32Z","updated_at":"2015-03-30T18:25:19Z","closed_at":null,"body":"We were running tests on kubernetes and encountered a split-brain issue, we are pouring through the logs and data to diagnose.  Is there a secure place to handoff data?  \r\n\r\netcd version 2.0.4+git  635e4db6d942cd84e8b613c37c6b47f8c06b94dd\r\n\r\n\r\nHere are the symptoms: \r\n\r\n\\#etcdctl ls /\r\n/coreos.com\r\n/fuckotest\r\n\r\ncurl -L http://host01-rack10:4001/v2/keys/ | python -mjson.tool\r\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\r\n                                 Dload  Upload   Total   Spent    Left  Speed\r\n100   194  100   194    0     0  29181      0 --:--:-- --:--:-- --:--:-- 32333\r\n{\r\n    \"action\": \"get\",\r\n    \"node\": {\r\n        \"dir\": true,\r\n        \"nodes\": [\r\n            {\r\n                \"createdIndex\": 5,\r\n                \"dir\": true,\r\n                \"key\": \"/coreos.com\",\r\n                \"modifiedIndex\": 5\r\n            },\r\n            {\r\n                \"createdIndex\": 3880163,\r\n                \"dir\": true,\r\n                \"key\": \"/fuckotest\",\r\n                \"modifiedIndex\": 3880163\r\n            }\r\n        ]\r\n    }\r\n}\r\n\r\n\\# curl -L http://host02-rack10:4001/v2/keys/ | python -mjson.tool\r\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\r\n                                 Dload  Upload   Total   Spent    Left  Speed\r\n100   189  100   189    0     0  30691      0 --:--:-- --:--:-- --:--:-- 37800\r\n{\r\n    \"action\": \"get\",\r\n    \"node\": {\r\n        \"dir\": true,\r\n        \"nodes\": [\r\n            {\r\n                \"createdIndex\": 5,\r\n                \"dir\": true,\r\n                \"key\": \"/registry\",\r\n                \"modifiedIndex\": 5\r\n            },\r\n            {\r\n                \"createdIndex\": 15588,\r\n                \"dir\": true,\r\n                \"key\": \"/coreos.com\",\r\n                \"modifiedIndex\": 15588\r\n            }\r\n        ]\r\n    }\r\n}\r\n\r\n\\# curl -L http://host17-rack11:4001/v2/keys/ | python -mjson.tool\r\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\r\n                                 Dload  Upload   Total   Spent    Left  Speed\r\n100   113  100   113    0     0  41061      0 --:--:-- --:--:-- --:--:-- 56500\r\n{\r\n    \"action\": \"get\",\r\n    \"node\": {\r\n        \"dir\": true,\r\n        \"nodes\": [\r\n            {\r\n                \"createdIndex\": 4,\r\n                \"dir\": true,\r\n                \"key\": \"/registry\",\r\n                \"modifiedIndex\": 4\r\n            }\r\n        ]\r\n    }\r\n}\r\n"},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/87780592","html_url":"https://github.com/coreos/etcd/issues/2592#issuecomment-87780592","issue_url":"https://api.github.com/repos/coreos/etcd/issues/2592","id":87780592,"user":{"login":"timothysc","id":169553,"avatar_url":"https://avatars.githubusercontent.com/u/169553?v=3","gravatar_id":"","url":"https://api.github.com/users/timothysc","html_url":"https://github.com/timothysc","followers_url":"https://api.github.com/users/timothysc/followers","following_url":"https://api.github.com/users/timothysc/following{/other_user}","gists_url":"https://api.github.com/users/timothysc/gists{/gist_id}","starred_url":"https://api.github.com/users/timothysc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timothysc/subscriptions","organizations_url":"https://api.github.com/users/timothysc/orgs","repos_url":"https://api.github.com/users/timothysc/repos","events_url":"https://api.github.com/users/timothysc/events{/privacy}","received_events_url":"https://api.github.com/users/timothysc/received_events","type":"User","site_admin":false},"created_at":"2015-03-30T18:25:19Z","updated_at":"2015-03-30T18:25:19Z","body":"That's weird, below are our configs. \r\n\r\nhost01-rack10.scale.openstack.engineering.redhat.com\r\n\r\nETCD_NAME=host01-rack10\r\nETCD_DATA_DIR=\"/var/lib/etcd/default.etcd\"\r\nETCD_LISTEN_PEER_URLS=\"http://host01-rack10.scale.openstack.engineering.redhat.com:2380,http://localhost:2380,http://host01-rack10.scale.openstack.engineering.redhat.com:7001,http://localhost:7001\"\r\nETCD_LISTEN_CLIENT_URLS=\"http://host01-rack10.scale.openstack.engineering.redhat.com:2379,http://localhost:2379,http://host01-rack10.scale.openstack.engineering.redhat.com:4001,http://localhost:4001\"\r\nETCD_INITIAL_ADVERTISE_PEER_URLS=\"http://host01-rack10.scale.openstack.engineering.redhat.com:2380,http://localhost:2380,http://host01-rack10.scale.openstack.engineering.redhat.com:7001,http://localhost:7001\"\r\nETCD_INITIAL_CLUSTER=\"host01-rack10=http://host01-rack10.scale.openstack.engineering.redhat.com:7001,host02-rack10=http://host02-rack10.scale.openstack.engineering.redhat.com:7001,host17-rack11=http://host17-rack11.scale.openstack.engineering.redhat.com:7001\"\r\nETCD_INITIAL_CLUSTER_STATE=\"new\"\r\nETCD_INITIAL_CLUSTER_TOKEN=\"etcd-cluster-1\"\r\nETCD_ADVERTISE_CLIENT_URLS=\"http://host01-rack10.scale.openstack.engineering.redhat.com:2379,http://localhost:2379,http://host01-rack10.scale.openstack.engineering.redhat.com:4001,http://localhost:4001\"\r\n\r\nhost02-rack10.scale.openstack.engineering.redhat.com\r\nETCD_NAME=host02-rack10\r\nETCD_DATA_DIR=\"/var/lib/etcd/default.etcd\"\r\nETCD_LISTEN_PEER_URLS=\"http://host02-rack10.scale.openstack.engineering.redhat.com:2380,http://localhost:2380,http://host02-rack10.scale.openstack.engineering.redhat.com:7001,http://localhost:7001\"\r\nETCD_LISTEN_CLIENT_URLS=\"http://host02-rack10.scale.openstack.engineering.redhat.com:2379,http://localhost:2379,http://host02-rack10.scale.openstack.engineering.redhat.com:4001,http://localhost:4001\"\r\nETCD_INITIAL_ADVERTISE_PEER_URLS=\"http://host02-rack10.scale.openstack.engineering.redhat.com:2380,http://localhost:2380,http://host02-rack10.scale.openstack.engineering.redhat.com:7001,http://localhost:7001\"\r\nETCD_INITIAL_CLUSTER=\"host01-rack10=http://host01-rack10.scale.openstack.engineering.redhat.com:7001,host02-rack10=http://host02-rack10.scale.openstack.engineering.redhat.com:7001,host17-rack11=http://host17-rack11.scale.openstack.engineering.redhat.com:7001\"\r\nETCD_INITIAL_CLUSTER_STATE=\"new\"\r\nETCD_INITIAL_CLUSTER_TOKEN=\"etcd-cluster-1\"\r\nETCD_ADVERTISE_CLIENT_URLS=\"http://host02-rack10.scale.openstack.engineering.redhat.com:2379,http://localhost:2379,http://host02-\r\nrack10.scale.openstack.engineering.redhat.com:4001,http://localhost:4001\"\r\n\r\nhost17-rack11.scale.openstack.engineering.redhat.com\r\nETCD_NAME=\"host17-rack11\"\r\nETCD_DATA_DIR=\"/var/lib/etcd/default.etcd\"\r\nETCD_LISTEN_PEER_URLS=\"http://host17-rack11.scale.openstack.engineering.redhat.com:2380,http://localhost:2380,http://host17-rack11.scale.openstack.engineering.redhat.com:7001,http://localhost:7001\"\r\nETCD_LISTEN_CLIENT_URLS=\"http://host17-rack11.scale.openstack.engineering.redhat.com:2379,http://localhost:2379,http://host17-rack11.scale.openstack.engineering.redhat.com:4001,http://localhost:4001\"\r\nETCD_INITIAL_ADVERTISE_PEER_URLS=\"http://host17-rack11.scale.openstack.engineering.redhat.com:2380,http://localhost:2380,http://host17-rack11.scale.openstack.engineering.redhat.com:7001,http://localhost:7001\"\r\nETCD_INITIAL_CLUSTER=\"host01-rack10=http://host01-rack10.scale.openstack.engineering.redhat.com:7001,host02-rack10=http://host02-rack10.scale.openstack.engineering.redhat.com:7001,host17-rack11=http://host17-rack11.scale.openstack.engineering.redhat.com:7001\"\r\nETCD_INITIAL_CLUSTER_STATE=\"new\"\r\nETCD_INITIAL_CLUSTER_TOKEN=\"etcd-cluster-1\"\r\nETCD_ADVERTISE_CLIENT_URLS=\"http://host17-rack11.scale.openstack.engineering.redhat.com:2379,http://localhost:2379,http://host17-rack11.scale.openstack.engineering.redhat.com:4001,http://localhost:4001\""}}