{"action":"closed","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16623","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16623/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16623/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16623/events","html_url":"https://github.com/kubernetes/kubernetes/issues/16623","id":114362665,"number":16623,"title":"e2e flake: Daemon set should run and stop complex daemon","user":{"login":"a-robinson","id":7085343,"avatar_url":"https://avatars.githubusercontent.com/u/7085343?v=3","gravatar_id":"","url":"https://api.github.com/users/a-robinson","html_url":"https://github.com/a-robinson","followers_url":"https://api.github.com/users/a-robinson/followers","following_url":"https://api.github.com/users/a-robinson/following{/other_user}","gists_url":"https://api.github.com/users/a-robinson/gists{/gist_id}","starred_url":"https://api.github.com/users/a-robinson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a-robinson/subscriptions","organizations_url":"https://api.github.com/users/a-robinson/orgs","repos_url":"https://api.github.com/users/a-robinson/repos","events_url":"https://api.github.com/users/a-robinson/events{/privacy}","received_events_url":"https://api.github.com/users/a-robinson/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/flake","name":"kind/flake","color":"f7c6c7"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P0","name":"priority/P0","color":"e11d21"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/control-plane","name":"team/control-plane","color":"d2b48c"}],"state":"closed","locked":false,"assignee":{"login":"mikedanese","id":1787169,"avatar_url":"https://avatars.githubusercontent.com/u/1787169?v=3","gravatar_id":"","url":"https://api.github.com/users/mikedanese","html_url":"https://github.com/mikedanese","followers_url":"https://api.github.com/users/mikedanese/followers","following_url":"https://api.github.com/users/mikedanese/following{/other_user}","gists_url":"https://api.github.com/users/mikedanese/gists{/gist_id}","starred_url":"https://api.github.com/users/mikedanese/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikedanese/subscriptions","organizations_url":"https://api.github.com/users/mikedanese/orgs","repos_url":"https://api.github.com/users/mikedanese/repos","events_url":"https://api.github.com/users/mikedanese/events{/privacy}","received_events_url":"https://api.github.com/users/mikedanese/received_events","type":"User","site_admin":false},"milestone":null,"comments":4,"created_at":"2015-10-30T21:55:27Z","updated_at":"2015-12-07T21:51:24Z","closed_at":"2015-12-07T21:51:24Z","body":"This has been flaking pretty often lately - in 5 out of the last 11 runs of kubernetes-e2e-gce-flaky, to be specific. I've copied the output of one such failure run below.\r\n\r\nSorry if there's already an issue tracking this, I couldn't find anything other than #14072, which is for a different test.\r\n\r\n@mikedanese \r\n\r\n```\r\n14:35:47 Daemon set \r\n14:35:47   should run and stop complex daemon\r\n14:35:47   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:181\r\n14:35:47 [BeforeEach] Daemon set\r\n14:35:47   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework.go:51\r\n14:35:47 STEP: Creating a kubernetes client\r\n14:35:47 >>> testContext.KubeConfig: /var/lib/jenkins/jobs/kubernetes-e2e-gce-flaky/workspace/.kube/config\r\n14:35:47 STEP: Building a namespace api object\r\n14:35:47 Oct 30 14:35:47.266: INFO: Waiting up to 2m0s for service account default to be provisioned in ns e2e-tests-daemonsets-40nfz\r\n14:35:47 Oct 30 14:35:47.268: INFO: Get service account default in ns e2e-tests-daemonsets-40nfz failed, ignoring for 2s: serviceaccounts \"default\" not found\r\n14:35:49 Oct 30 14:35:49.271: INFO: Service account default in ns e2e-tests-daemonsets-40nfz with secrets found. (2.005191042s)\r\n14:35:49 STEP: Waiting for a default service account to be provisioned in namespace\r\n14:35:49 Oct 30 14:35:49.271: INFO: Waiting up to 2m0s for service account default to be provisioned in ns e2e-tests-daemonsets-40nfz\r\n14:35:49 Oct 30 14:35:49.273: INFO: Service account default in ns e2e-tests-daemonsets-40nfz with secrets found. (1.930327ms)\r\n14:35:49 [BeforeEach] Daemon set\r\n14:35:49   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:71\r\n14:35:55 [It] should run and stop complex daemon\r\n14:35:55   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:181\r\n14:35:55 Oct 30 14:35:55.287: INFO: Creating daemon with a node selector daemon-set\r\n14:35:55 STEP: Initially, daemon pods should not be running on any nodes.\r\n14:35:57 Oct 30 14:35:57.297: INFO: nodesToPodCount: map[string]int{}\r\n14:35:57 STEP: Change label of node, check that daemon pod is launched.\r\n14:36:01 Oct 30 14:36:01.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:03 Oct 30 14:36:03.347: INFO: nodesToPodCount: map[string]int{}\r\n14:36:05 Oct 30 14:36:05.319: INFO: nodesToPodCount: map[string]int{}\r\n14:36:07 Oct 30 14:36:07.319: INFO: nodesToPodCount: map[string]int{}\r\n14:36:09 Oct 30 14:36:09.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:11 Oct 30 14:36:11.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:13 Oct 30 14:36:13.350: INFO: nodesToPodCount: map[string]int{}\r\n14:36:15 Oct 30 14:36:15.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:17 Oct 30 14:36:17.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:19 Oct 30 14:36:19.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:21 Oct 30 14:36:21.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:23 Oct 30 14:36:23.347: INFO: nodesToPodCount: map[string]int{}\r\n14:36:25 Oct 30 14:36:25.319: INFO: nodesToPodCount: map[string]int{}\r\n14:36:27 Oct 30 14:36:27.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:29 Oct 30 14:36:29.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:31 Oct 30 14:36:31.350: INFO: nodesToPodCount: map[string]int{}\r\n14:36:33 Oct 30 14:36:33.352: INFO: nodesToPodCount: map[string]int{}\r\n14:36:35 Oct 30 14:36:35.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:37 Oct 30 14:36:37.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:39 Oct 30 14:36:39.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:41 Oct 30 14:36:41.319: INFO: nodesToPodCount: map[string]int{}\r\n14:36:43 Oct 30 14:36:43.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:45 Oct 30 14:36:45.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:47 Oct 30 14:36:47.319: INFO: nodesToPodCount: map[string]int{}\r\n14:36:49 Oct 30 14:36:49.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:51 Oct 30 14:36:51.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:53 Oct 30 14:36:53.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:55 Oct 30 14:36:55.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:57 Oct 30 14:36:57.318: INFO: nodesToPodCount: map[string]int{}\r\n14:36:59 Oct 30 14:36:59.319: INFO: nodesToPodCount: map[string]int{}\r\n14:36:59 Oct 30 14:36:59.321: INFO: nodesToPodCount: map[string]int{}\r\n14:36:59 [AfterEach] Daemon set\r\n14:36:59   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:56\r\n14:37:05 [AfterEach] Daemon set\r\n14:37:05   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework.go:52\r\n14:37:05 STEP: Collecting events from namespace \"e2e-tests-daemonsets-40nfz\".\r\n14:37:05 Oct 30 14:37:05.380: INFO: POD                                          NODE                   PHASE    GRACE  CONDITIONS\r\n14:37:05 Oct 30 14:37:05.380: INFO: elasticsearch-logging-v1-jmu9g               e2e-flaky-minion-lojz  Running         [{Ready True 0001-01-01 00:00:00 +0000 UTC 2015-10-30 14:34:21 -0700 PDT  }]\r\n14:37:05 Oct 30 14:37:05.380: INFO: elasticsearch-logging-v1-z9x37               e2e-flaky-minion-ray5  Running         [{Ready True 0001-01-01 00:00:00 +0000 UTC 2015-10-30 14:28:54 -0700 PDT  }]\r\n14:37:05 Oct 30 14:37:05.380: INFO: fluentd-elasticsearch-e2e-flaky-minion-7zwb  e2e-flaky-minion-7zwb  Running         [{Ready True 0001-01-01 00:00:00 +0000 UTC 2015-10-30 14:35:48 -0700 PDT  }]\r\n14:37:05 Oct 30 14:37:05.380: INFO: fluentd-elasticsearch-e2e-flaky-minion-lojz  e2e-flaky-minion-lojz  Running         [{Ready True 0001-01-01 00:00:00 +0000 UTC 2015-10-30 14:28:14 -0700 PDT  }]\r\n14:37:05 Oct 30 14:37:05.380: INFO: fluentd-elasticsearch-e2e-flaky-minion-ray5  e2e-flaky-minion-ray5  Running         [{Ready True 0001-01-01 00:00:00 +0000 UTC 2015-10-30 14:28:19 -0700 PDT  }]\r\n14:37:05 Oct 30 14:37:05.380: INFO: heapster-v10-crc4j                           e2e-flaky-minion-lojz  Running         [{Ready True 0001-01-01 00:00:00 +0000 UTC 2015-10-30 14:34:03 -0700 PDT  }]\r\n14:37:05 Oct 30 14:37:05.380: INFO: kibana-logging-v1-e6e7p                      e2e-flaky-minion-ray5  Running         [{Ready True 0001-01-01 00:00:00 +0000 UTC 2015-10-30 14:34:19 -0700 PDT  }]\r\n14:37:05 Oct 30 14:37:05.380: INFO: kube-dns-v9-dzbwq                            e2e-flaky-minion-ray5  Running         [{Ready True 0001-01-01 00:00:00 +0000 UTC 2015-10-30 14:28:54 -0700 PDT  }]\r\n14:37:05 Oct 30 14:37:05.380: INFO: kube-ui-v3-hx16m                             e2e-flaky-minion-lojz  Running         [{Ready True 0001-01-01 00:00:00 +0000 UTC 2015-10-30 14:28:21 -0700 PDT  }]\r\n14:37:05 Oct 30 14:37:05.380: INFO: l7-lb-controller-nvr5o                       e2e-flaky-minion-ray5  Running         [{Ready True 0001-01-01 00:00:00 +0000 UTC 2015-10-30 14:28:53 -0700 PDT  }]\r\n14:37:05 Oct 30 14:37:05.380: INFO: monitoring-influxdb-grafana-v2-76koa         e2e-flaky-minion-lojz  Running         [{Ready True 0001-01-01 00:00:00 +0000 UTC 2015-10-30 14:28:42 -0700 PDT  }]\r\n14:37:05 Oct 30 14:37:05.380: INFO: \r\n14:37:05 Oct 30 14:37:05.380: INFO: Waiting up to 1m0s for all nodes to be ready\r\n14:37:05 Oct 30 14:37:05.384: INFO: Node e2e-flaky-minion-7zwb condition 1/2: type: Ready, status: True, reason: \"KubeletReady\", message: \"kubelet is posting ready status\", last transition time: 2015-10-30 14:35:08 -0700 PDT\r\n14:37:05 Oct 30 14:37:05.384: INFO: Successfully found node e2e-flaky-minion-7zwb readiness to be true\r\n14:37:05 Oct 30 14:37:05.384: INFO: Node e2e-flaky-minion-lojz condition 1/2: type: Ready, status: True, reason: \"KubeletReady\", message: \"kubelet is posting ready status\", last transition time: 2015-10-30 14:27:50 -0700 PDT\r\n14:37:05 Oct 30 14:37:05.384: INFO: Successfully found node e2e-flaky-minion-lojz readiness to be true\r\n14:37:05 Oct 30 14:37:05.384: INFO: Node e2e-flaky-minion-ray5 condition 1/2: type: Ready, status: True, reason: \"KubeletReady\", message: \"kubelet is posting ready status\", last transition time: 2015-10-30 14:27:54 -0700 PDT\r\n14:37:05 Oct 30 14:37:05.384: INFO: Successfully found node e2e-flaky-minion-ray5 readiness to be true\r\n14:37:05 STEP: Destroying namespace \"e2e-tests-daemonsets-40nfz\" for this suite.\r\n14:37:10 \r\n14:37:10 • Failure [83.142 seconds]\r\n14:37:10 Daemon set\r\n14:37:10 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:182\r\n14:37:10   should run and stop complex daemon [It]\r\n14:37:10   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:181\r\n14:37:10 \r\n14:37:10   error waiting for daemon pods to be running on new nodes\r\n14:37:10   Expected error:\r\n14:37:10       <*errors.errorString | 0xc20814d4d0>: {\r\n14:37:10           s: \"timed out waiting for the condition\",\r\n14:37:10       }\r\n14:37:10       timed out waiting for the condition\r\n14:37:10   not to have occurred\r\n14:37:10 \r\n14:37:10   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:170\r\n```"}}