{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16699","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16699/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16699/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16699/events","html_url":"https://github.com/kubernetes/kubernetes/issues/16699","id":114716664,"number":16699,"title":"Watch requests for specific resources fail to return data (sometimes)","user":{"login":"evanphx","id":7,"avatar_url":"https://avatars.githubusercontent.com/u/7?v=3","gravatar_id":"","url":"https://api.github.com/users/evanphx","html_url":"https://github.com/evanphx","followers_url":"https://api.github.com/users/evanphx/followers","following_url":"https://api.github.com/users/evanphx/following{/other_user}","gists_url":"https://api.github.com/users/evanphx/gists{/gist_id}","starred_url":"https://api.github.com/users/evanphx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evanphx/subscriptions","organizations_url":"https://api.github.com/users/evanphx/orgs","repos_url":"https://api.github.com/users/evanphx/repos","events_url":"https://api.github.com/users/evanphx/events{/privacy}","received_events_url":"https://api.github.com/users/evanphx/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/CSI","name":"team/CSI","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-11-03T00:35:57Z","updated_at":"2015-11-03T00:52:22Z","closed_at":null,"body":"* Given an service named `jpl-input`\r\n* A request (call it request A) is issued: `curl localhost:8001/api/v1/watch/namespaces/default/endpoints/jpl-input`.\r\n* A request (call it request B) is issued: `curl localhost:8001/api/v1/watch/namespaces/default/endpoints`\r\n* A pod that matches the service `jpl-input` is then started (either directly or via a rolling-update).\r\n* Request B begins getting data: `{\"type\":\"MODIFIED\",\"object\":{\"kind\":\"Endpoints\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"jpl-input\",\"namespace\":\"default\",\"sel...`\r\n* Request A stays silent and never returns any data.\r\n\r\nThe expectation is that request A is sent updates about the endpoint.\r\n\r\n\r\nHere is the tcp session between apiserver and etcd for the processing of request A:\r\n\r\n```\r\nroot@ip-172-20-0-9:~# cat watch-on-specific-name \r\ninput: watch.dump\r\nfilter: (ip or ip6) and ( port 52336 )\r\nmatch: .\r\n#######\r\nT 127.0.0.1:52336 -> 127.0.0.1:4001 [AP]\r\n  GET /v2/keys/registry/services/endpoints/default/jpl-input?quorum=false&recursive=false&sorted=false HTTP/1.1..Host: 127.0.0.1:4\r\n  001..User-Agent: Go 1.1 package http..Accept-Encoding: gzip....                                                                 \r\n#\r\nT 127.0.0.1:4001 -> 127.0.0.1:52336 [AP]\r\n  HTTP/1.1 200 OK..Content-Type: application/json..X-Etcd-Cluster-Id: 7e27652122e8b2ae..X-Etcd-Index: 1809352..X-Raft-Index: 13467\r\n  978..X-Raft-Term: 5..Date: Tue, 03 Nov 2015 00:16:35 GMT..Content-Length: 808....{\"action\":\"get\",\"node\":{\"key\":\"/registry/servic\r\n  es/endpoints/default/jpl-input\",\"value\":\"{\\\"kind\\\":\\\"Endpoints\\\",\\\"apiVersion\\\":\\\"v1\\\",\\\"metadata\\\":{\\\"name\\\":\\\"jpl-input\\\",\\\"na\r\n  mespace\\\":\\\"default\\\",\\\"selfLink\\\":\\\"/api/v1/namespaces/default/endpoints/jpl-input\\\",\\\"uid\\\":\\\"c2b1ff23-6eb8-11e5-945c-024ba4ed\r\n  8969\\\",\\\"resourceVersion\\\":\\\"1809296\\\",\\\"creationTimestamp\\\":\\\"2015-10-09T19:05:52Z\\\",\\\"labels\\\":{\\\"name\\\":\\\"jpl-input\\\"}},\\\"sub\r\n  sets\\\":[{\\\"addresses\\\":[{\\\"ip\\\":\\\"10.244.0.65\\\",\\\"targetRef\\\":{\\\"kind\\\":\\\"Pod\\\",\\\"namespace\\\":\\\"default\\\",\\\"name\\\":\\\"formatter-0\r\n  50683415242a28fb581d52fe0e42834-7zcpo\\\",\\\"uid\\\":\\\"cd2e2900-81bf-11e5-9bf3-024ba4ed8969\\\",\\\"resourceVersion\\\":\\\"1809295\\\"}}],\\\"po\r\n  rts\\\":[{\\\"name\\\":\\\"jpl-input\\\",\\\"port\\\":9101,\\\"protocol\\\":\\\"TCP\\\"}]}]}\",\"modifiedIndex\":1809318,\"createdIndex\":1172435}}.       \r\n##\r\nT 127.0.0.1:52336 -> 127.0.0.1:4001 [AP]\r\n  GET /v2/keys/registry/services/endpoints/default/jpl-input?wait=true&waitIndex=1809353 HTTP/1.1..Host: 127.0.0.1:4001..User-Agen\r\n  t: Go 1.1 package http..Accept-Encoding: gzip....                                                                               \r\n#\r\nT 127.0.0.1:4001 -> 127.0.0.1:52336 [AP]\r\n  HTTP/1.1 200 OK..Content-Type: application/json..X-Etcd-Cluster-Id: 7e27652122e8b2ae..X-Etcd-Index: 1809352..X-Raft-Index: 13467\r\n  978..X-Raft-Term: 5..Date: Tue, 03 Nov 2015 00:16:35 GMT..Transfer-Encoding: chunked....                                        \r\n########################################\r\nT 127.0.0.1:4001 -> 127.0.0.1:52336 [AP]\r\n  731..{\"action\":\"compareAndSwap\",\"node\":{\"key\":\"/registry/services/endpoints/default/jpl-input\",\"value\":\"{\\\"kind\\\":\\\"Endpoints\\\",\r\n  \\\"apiVersion\\\":\\\"v1\\\",\\\"metadata\\\":{\\\"name\\\":\\\"jpl-input\\\",\\\"namespace\\\":\\\"default\\\",\\\"selfLink\\\":\\\"/api/v1/namespaces/default/e\r\n  ndpoints/jpl-input\\\",\\\"uid\\\":\\\"c2b1ff23-6eb8-11e5-945c-024ba4ed8969\\\",\\\"resourceVersion\\\":\\\"1809318\\\",\\\"creationTimestamp\\\":\\\"20\r\n  15-10-09T19:05:52Z\\\",\\\"labels\\\":{\\\"name\\\":\\\"jpl-input\\\"}},\\\"subsets\\\":[{\\\"addresses\\\":[{\\\"ip\\\":\\\"10.244.0.65\\\",\\\"targetRef\\\":{\\\"\r\n  kind\\\":\\\"Pod\\\",\\\"namespace\\\":\\\"default\\\",\\\"name\\\":\\\"formatter-050683415242a28fb581d52fe0e42834-7zcpo\\\",\\\"uid\\\":\\\"cd2e2900-81bf-1\r\n  1e5-9bf3-024ba4ed8969\\\",\\\"resourceVersion\\\":\\\"1809295\\\"}},{\\\"ip\\\":\\\"10.244.2.85\\\",\\\"targetRef\\\":{\\\"kind\\\":\\\"Pod\\\",\\\"namespace\\\":\r\n  \\\"default\\\",\\\"name\\\":\\\"formatter-050683415242a28fb581d52fe0e42834-ucva4\\\",\\\"uid\\\":\\\"29dc945a-81c0-11e5-9bf3-024ba4ed8969\\\",\\\"res\r\n  ourceVersion\\\":\\\"1809378\\\"}}],\\\"ports\\\":[{\\\"name\\\":\\\"jpl-input\\\",\\\"port\\\":9101,\\\"protocol\\\":\\\"TCP\\\"}]}]}\",\"modifiedIndex\":180937\r\n  9,\"createdIndex\":1172435},\"prevNode\":{\"key\":\"/registry/services/endpoints/default/jpl-input\",\"value\":\"{\\\"kind\\\":\\\"Endpoints\\\",\\\"\r\n  apiVersion\\\":\\\"v1\\\",\\\"metadata\\\":{\\\"name\\\":\\\"jpl-input\\\",\\\"namespace\\\":\\\"default\\\",\\\"selfLink\\\":\\\"/api/v1/namespaces/default/end\r\n  points/jpl-input\\\",\\\"uid\\\":\\\"c2b1ff23-6eb8-11e5-945c-024ba4ed8969\\\",\\\"resourceVersion\\\":\\\"1809296\\\",\\\"creationTimestamp\\\":\\\"2015\r\n  -10-09T19:05:52Z\\\",\\\"labels\\\":{\\\"name\\\":\\\"jpl-input\\\"}},\\\"subsets\\\":[{\\\"addresses\\\":[{\\\"ip\\\":\\\"10.244.0.65\\\",\\\"targetRef\\\":{\\\"ki\r\n  nd\\\":\\\"Pod\\\",\\\"namespace\\\":\\\"default\\\",\\\"name\\\":\\\"formatter-050683415242a28fb581d52fe0e42834-7zcpo\\\",\\\"uid\\\":\\\"cd2e2900-81bf-11e\r\n  5-9bf3-024ba4ed8969\\\",\\\"resourceVersion\\\":\\\"1809295\\\"}}],\\\"ports\\\":[{\\\"name\\\":\\\"jpl-input\\\",\\\"port\\\":9101,\\\"protocol\\\":\\\"TCP\\\"}]\r\n  }]}\",\"modifiedIndex\":1809318,\"createdIndex\":1172435}}...0....                                                                   \r\n##\r\nT 127.0.0.1:52336 -> 127.0.0.1:4001 [AP]\r\n  GET /v2/keys/registry/services/endpoints/default/jpl-input?wait=true&waitIndex=1809380 HTTP/1.1..Host: 127.0.0.1:4001..User-Agen\r\n  t: Go 1.1 package http..Accept-Encoding: gzip....                                                                               \r\n#\r\nT 127.0.0.1:4001 -> 127.0.0.1:52336 [AP]\r\n  HTTP/1.1 200 OK..Content-Type: application/json..X-Etcd-Cluster-Id: 7e27652122e8b2ae..X-Etcd-Index: 1809379..X-Raft-Index: 13468\r\n  044..X-Raft-Term: 5..Date: Tue, 03 Nov 2015 00:16:54 GMT..Transfer-Encoding: chunked....                                        \r\n####################################################################################################################\r\nT 127.0.0.1:4001 -> 127.0.0.1:52336 [AP]\r\n  731..{\"action\":\"compareAndSwap\",\"node\":{\"key\":\"/registry/services/endpoints/default/jpl-input\",\"value\":\"{\\\"kind\\\":\\\"Endpoints\\\",\r\n  \\\"apiVersion\\\":\\\"v1\\\",\\\"metadata\\\":{\\\"name\\\":\\\"jpl-input\\\",\\\"namespace\\\":\\\"default\\\",\\\"selfLink\\\":\\\"/api/v1/namespaces/default/e\r\n  ndpoints/jpl-input\\\",\\\"uid\\\":\\\"c2b1ff23-6eb8-11e5-945c-024ba4ed8969\\\",\\\"resourceVersion\\\":\\\"1809379\\\",\\\"creationTimestamp\\\":\\\"20\r\n  15-10-09T19:05:52Z\\\",\\\"labels\\\":{\\\"name\\\":\\\"jpl-input\\\"}},\\\"subsets\\\":[{\\\"addresses\\\":[{\\\"ip\\\":\\\"10.244.2.85\\\",\\\"targetRef\\\":{\\\"\r\n  kind\\\":\\\"Pod\\\",\\\"namespace\\\":\\\"default\\\",\\\"name\\\":\\\"formatter-050683415242a28fb581d52fe0e42834-ucva4\\\",\\\"uid\\\":\\\"29dc945a-81c0-1\r\n  1e5-9bf3-024ba4ed8969\\\",\\\"resourceVersion\\\":\\\"1809378\\\"}}],\\\"ports\\\":[{\\\"name\\\":\\\"jpl-input\\\",\\\"port\\\":9101,\\\"protocol\\\":\\\"TCP\\\"\r\n  }]}]}\",\"modifiedIndex\":1809401,\"createdIndex\":1172435},\"prevNode\":{\"key\":\"/registry/services/endpoints/default/jpl-input\",\"value\r\n  \":\"{\\\"kind\\\":\\\"Endpoints\\\",\\\"apiVersion\\\":\\\"v1\\\",\\\"metadata\\\":{\\\"name\\\":\\\"jpl-input\\\",\\\"namespace\\\":\\\"default\\\",\\\"selfLink\\\":\\\"/\r\n  api/v1/namespaces/default/endpoints/jpl-input\\\",\\\"uid\\\":\\\"c2b1ff23-6eb8-11e5-945c-024ba4ed8969\\\",\\\"resourceVersion\\\":\\\"1809318\\\"\r\n  ,\\\"creationTimestamp\\\":\\\"2015-10-09T19:05:52Z\\\",\\\"labels\\\":{\\\"name\\\":\\\"jpl-input\\\"}},\\\"subsets\\\":[{\\\"addresses\\\":[{\\\"ip\\\":\\\"10.2\r\n  44.0.65\\\",\\\"targetRef\\\":{\\\"kind\\\":\\\"Pod\\\",\\\"namespace\\\":\\\"default\\\",\\\"name\\\":\\\"formatter-050683415242a28fb581d52fe0e42834-7zcpo\\\r\n  \",\\\"uid\\\":\\\"cd2e2900-81bf-11e5-9bf3-024ba4ed8969\\\",\\\"resourceVersion\\\":\\\"1809295\\\"}},{\\\"ip\\\":\\\"10.244.2.85\\\",\\\"targetRef\\\":{\\\"ki\r\n  nd\\\":\\\"Pod\\\",\\\"namespace\\\":\\\"default\\\",\\\"name\\\":\\\"formatter-050683415242a28fb581d52fe0e42834-ucva4\\\",\\\"uid\\\":\\\"29dc945a-81c0-11e\r\n  5-9bf3-024ba4ed8969\\\",\\\"resourceVersion\\\":\\\"1809378\\\"}}],\\\"ports\\\":[{\\\"name\\\":\\\"jpl-input\\\",\\\"port\\\":9101,\\\"protocol\\\":\\\"TCP\\\"}]\r\n  }]}\",\"modifiedIndex\":1809379,\"createdIndex\":1172435}}...0....                                                                   \r\n##\r\nT 127.0.0.1:52336 -> 127.0.0.1:4001 [AP]\r\n  GET /v2/keys/registry/services/endpoints/default/jpl-input?wait=true&waitIndex=1809402 HTTP/1.1..Host: 127.0.0.1:4001..User-Agen\r\n  t: Go 1.1 package http..Accept-Encoding: gzip....                                                                               \r\n#\r\nT 127.0.0.1:4001 -> 127.0.0.1:52336 [AP]\r\n  HTTP/1.1 200 OK..Content-Type: application/json..X-Etcd-Cluster-Id: 7e27652122e8b2ae..X-Etcd-Index: 1809401..X-Raft-Index: 13468\r\n  179..X-Raft-Term: 5..Date: Tue, 03 Nov 2015 00:17:50 GMT..Transfer-Encoding: chunked....                                        \r\n###################exit\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/153207167","html_url":"https://github.com/kubernetes/kubernetes/issues/16699#issuecomment-153207167","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16699","id":153207167,"user":{"login":"lavalamp","id":647318,"avatar_url":"https://avatars.githubusercontent.com/u/647318?v=3","gravatar_id":"","url":"https://api.github.com/users/lavalamp","html_url":"https://github.com/lavalamp","followers_url":"https://api.github.com/users/lavalamp/followers","following_url":"https://api.github.com/users/lavalamp/following{/other_user}","gists_url":"https://api.github.com/users/lavalamp/gists{/gist_id}","starred_url":"https://api.github.com/users/lavalamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lavalamp/subscriptions","organizations_url":"https://api.github.com/users/lavalamp/orgs","repos_url":"https://api.github.com/users/lavalamp/repos","events_url":"https://api.github.com/users/lavalamp/events{/privacy}","received_events_url":"https://api.github.com/users/lavalamp/received_events","type":"User","site_admin":false},"created_at":"2015-11-03T00:52:22Z","updated_at":"2015-11-03T00:52:22Z","body":"Recommend switching to `localhost:8001/api/v1/namespaces/default/endpoints?watch=true&metadata.name=jpl-input`, although I'm not sure off the top of my head why what you're doing isn't working."}}