{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/13129","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/13129/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/13129/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/13129/events","html_url":"https://github.com/kubernetes/kubernetes/issues/13129","id":102958853,"number":13129,"title":"google-fluentd using a lot of memory which got killed by kernel every 2~3 secs","user":{"login":"pencilcheck","id":116532,"avatar_url":"https://avatars.githubusercontent.com/u/116532?v=3","gravatar_id":"","url":"https://api.github.com/users/pencilcheck","html_url":"https://github.com/pencilcheck","followers_url":"https://api.github.com/users/pencilcheck/followers","following_url":"https://api.github.com/users/pencilcheck/following{/other_user}","gists_url":"https://api.github.com/users/pencilcheck/gists{/gist_id}","starred_url":"https://api.github.com/users/pencilcheck/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pencilcheck/subscriptions","organizations_url":"https://api.github.com/users/pencilcheck/orgs","repos_url":"https://api.github.com/users/pencilcheck/repos","events_url":"https://api.github.com/users/pencilcheck/events{/privacy}","received_events_url":"https://api.github.com/users/pencilcheck/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-08-25T06:29:09Z","updated_at":"2015-08-25T16:17:33Z","closed_at":null,"body":"One of the compute engine instance created by the container clusters has a fluentd that keeps crashing and using over 100% CPU. \r\n\r\nDisplaying the serial-port log of the node reveals that the process got killed by kernel cause it uses a lot of memory\r\nhttps://gist.github.com/pencilcheck/4d0700869fdd435e4593\r\n\r\nAnd the reason why it is using a lot of memory is unknown because there is no log when I try to inspect it via `kubectl`\r\n\r\n`kubectl get pods --namespace=kube-system then kubect logs <fluentd pod> --namespace=kube-system`\r\n\r\nThere are only 2 pods that belongs to the application, the rest are set up by the container clusters. Those 2 pods are both using less than 1% of memory I believe, and very low CPU usage.\r\n\r\n```\r\nName:                   gke-demo-7140bbcf-node-xzxu\r\nLabels:                 kubernetes.io/hostname=gke-demo-7140bbcf-node-xzxu\r\nCreationTimestamp:      Wed, 05 Aug 2015 14:20:20 -0700\r\nConditions:\r\n  Type          Status  LastHeartbeatTime                       LastTransitionTime                      Reason            Message\r\n  Ready         True    Mon, 24 Aug 2015 23:26:32 -0700         Wed, 05 Aug 2015 14:20:50 -0700         kubelet is posting ready status\r\nAddresses:      10.240.239.104,162.222.179.164\r\nCapacity:\r\n cpu:           1\r\n memory:        3800808Ki\r\n pods:          100\r\nVersion:\r\n Kernel Version:                3.16.0-0.bpo.4-amd64\r\n OS Image:                      Debian GNU/Linux 7 (wheezy)\r\n Container Runtime Version:     docker://Unknown\r\n Kubelet Version:               v0.21.4\r\n Kube-Proxy Version:            v0.21.4\r\nPodCIDR:                        10.128.1.0/24\r\nExternalID:                     12523289726931493693\r\nPods:                           (4 in total)\r\n  Namespace                     Name\r\n  default                       mailer-5379238f9b83cc212373e927b364639f-dw8pm\r\n  default                       nginx-466af8af4fb67dff39fed0240230dc30-vqe19\r\n  kube-system                   fluentd-cloud-logging-gke-demo-7140bbcf-node-xzxu\r\n  kube-system                   kube-dns-v8-1qg0o\r\nNo events.\r\n```\r\n\r\nThe node is still online, I thought leaving it up would be helpful so one google engineer can ssh in and debug, but it would be nice if I don't have to leave it up, so maybe I should kill the node and restart a new one, but that would not be helpful as this might happen again."},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/134657042","html_url":"https://github.com/kubernetes/kubernetes/issues/13129#issuecomment-134657042","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/13129","id":134657042,"user":{"login":"yujuhong","id":10524058,"avatar_url":"https://avatars.githubusercontent.com/u/10524058?v=3","gravatar_id":"","url":"https://api.github.com/users/yujuhong","html_url":"https://github.com/yujuhong","followers_url":"https://api.github.com/users/yujuhong/followers","following_url":"https://api.github.com/users/yujuhong/following{/other_user}","gists_url":"https://api.github.com/users/yujuhong/gists{/gist_id}","starred_url":"https://api.github.com/users/yujuhong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yujuhong/subscriptions","organizations_url":"https://api.github.com/users/yujuhong/orgs","repos_url":"https://api.github.com/users/yujuhong/repos","events_url":"https://api.github.com/users/yujuhong/events{/privacy}","received_events_url":"https://api.github.com/users/yujuhong/received_events","type":"User","site_admin":false},"created_at":"2015-08-25T16:17:32Z","updated_at":"2015-08-25T16:17:32Z","body":"cc @kubernetes/goog-gke "}}