{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/10282","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/10282/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/10282/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/10282/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/10282","id":90624850,"number":10282,"title":"Scalability tests failing on Jenkins due to non starting pods","user":{"login":"wojtek-t","id":10743879,"avatar_url":"https://avatars.githubusercontent.com/u/10743879?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/area/performance","name":"area/performance","color":"d4c5f9"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/area/reliability","name":"area/reliability","color":"a10d01"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/master","name":"team/master","color":"d2b48c"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/6","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/milestones/v1.0","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/6/labels","id":969317,"number":6,"title":"v1.0","description":"Issues that need to be resolved for v1.0 of Kubernetes.","creator":{"login":"goltermann","id":9358478,"avatar_url":"https://avatars.githubusercontent.com/u/9358478?v=3","gravatar_id":"","url":"https://api.github.com/users/goltermann","html_url":"https://github.com/goltermann","followers_url":"https://api.github.com/users/goltermann/followers","following_url":"https://api.github.com/users/goltermann/following{/other_user}","gists_url":"https://api.github.com/users/goltermann/gists{/gist_id}","starred_url":"https://api.github.com/users/goltermann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/goltermann/subscriptions","organizations_url":"https://api.github.com/users/goltermann/orgs","repos_url":"https://api.github.com/users/goltermann/repos","events_url":"https://api.github.com/users/goltermann/events{/privacy}","received_events_url":"https://api.github.com/users/goltermann/received_events","type":"User","site_admin":false},"open_issues":134,"closed_issues":550,"state":"open","created_at":"2015-02-06T20:27:16Z","updated_at":"2015-07-01T07:28:56Z","due_on":null,"closed_at":null},"comments":7,"created_at":"2015-06-24T09:35:30Z","updated_at":"2015-07-01T07:47:14Z","closed_at":null,"body":"There were a lot of failures in our scalability tests during last day (more than 50%). There seem to be two main reasons:\r\n- timeouts while waiting for deleting the previous namespace - this should be fixed by #10225 \r\n- some pods not starting\r\n\r\nI started looking deeper into those non-starting pods. Here is an example that contributed to the failure:\r\n- Jenkins scalability run 1539 - Density test\r\n- we are looking only on pods from \"e2e-tests-density-xyfx4\" namespace\r\n- look at the node \"e2e-scalability-minion-k2n5\"\r\n- scheduler scheduled 10 pods onto that node\r\n- it seems that at most 7 pods were actually started (7 logs \"Need to restart pod infra container ...\")\r\n- then there is a log: \"GET /healthz: (928.173627ms) 500\"\r\n- after that all the pods from that node were killed: \"killing unwanted pod\"\r\n\r\nLinks to logs:\r\n- Jenkins: http://kubekins.dls.corp.google.com/job/kubernetes-e2e-gce-scalability/1539/consoleFull\r\n- kubelet logs: https://storage.googleapis.com/kubernetes-jenkins/logs/kubernetes-e2e-gce-scalability/1539/_artifacts/104.197.100.6%3A22-kubelet.log\r\n- scheduler logs: https://storage.googleapis.com/kubernetes-jenkins/logs/kubernetes-e2e-gce-scalability/1539/_artifacts/104.154.56.51%3A22-kube-scheduler.log\r\n\r\nTo summarize, my hypothesis is that:\r\n- Kubelet syncPods seems to hang in this example\r\n- then I don't see any log saying that Kubelet was restarted, but it again starts to process thing\r\n- however, due to the reason I don't understand it kills all the pods that were running there\r\n\r\nI think that if the Kubelet didn't kill all the pods after again syncing pods, this wouldn't cause test failures. But anyway - there seem to be some problem here.\r\n\r\n@dchen1107 @yujuhong - can you please explain what exactly is happening there? I think that this is similar to what was observed in #9895, but I don't understand why all the pods are being killed.\r\n\r\ncc @fgrzadkowski @piosz @davidopp @bprashanth "},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/117515034","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/10282#issuecomment-117515034","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/10282","id":117515034,"user":{"login":"wojtek-t","id":10743879,"avatar_url":"https://avatars.githubusercontent.com/u/10743879?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false},"created_at":"2015-07-01T07:47:14Z","updated_at":"2015-07-01T07:47:14Z","body":"Yes - this is still an issue. The failure with exactly the same logs just happened tonight:\r\nhttp://kubekins.dls.corp.google.com/job/kubernetes-e2e-gce-scalability/1649/consoleFull\r\n\r\nFor the reference:\r\n- the node that had problems: e2e-scalability-minion-bv49 (104.154.79.113-22-kubelet.log in GCS)\r\n- again there are logs like this:\r\n\r\n        2814 I0701 05:44:17.407515   18374 container.go:369] Failed to update stats for container \"/2e20533074fb351801e684c5393474f9005f4ba8b450bb664c421e5837d19216\": du command failed on /var/lib/do     cker/aufs/diff/2e20533074fb351801e684c5393474f9005f4ba8b450bb664c421e5837d19216 with output  - fork/exec /usr/bin/du: cannot allocate memory, continuing to push stats\r\n"}}