{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17929","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17929/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17929/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17929/events","html_url":"https://github.com/kubernetes/kubernetes/issues/17929","id":119428677,"number":17929,"title":"Google Fluentd-Cloud-Logging pods are stuck in Terminating state","user":{"login":"paralin","id":534887,"avatar_url":"https://avatars.githubusercontent.com/u/534887?v=3","gravatar_id":"","url":"https://api.github.com/users/paralin","html_url":"https://github.com/paralin","followers_url":"https://api.github.com/users/paralin/followers","following_url":"https://api.github.com/users/paralin/following{/other_user}","gists_url":"https://api.github.com/users/paralin/gists{/gist_id}","starred_url":"https://api.github.com/users/paralin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/paralin/subscriptions","organizations_url":"https://api.github.com/users/paralin/orgs","repos_url":"https://api.github.com/users/paralin/repos","events_url":"https://api.github.com/users/paralin/events{/privacy}","received_events_url":"https://api.github.com/users/paralin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-11-30T06:36:55Z","updated_at":"2015-11-30T20:14:41Z","closed_at":null,"body":"My Google Cloud fluentd instances are stuck in \"terminating\" state under the kube-system namespace for over four hours now:\r\n\r\n```\r\nfluentd-cloud-logging-k8s-yasp-master        1/1       Terminating   0          1d\r\nfluentd-cloud-logging-k8s-yasp-minion-p5pm   1/1       Terminating   0          22h\r\nfluentd-cloud-logging-k8s-yasp-minion-szrb   1/1       Terminating   0          3h\r\n```\r\n\r\nI've tried deleting the pods directly as well and they stay in this state. If I try to view logs it says the pods do not exist. \r\n\r\nThey were not logging to the Google Logging before either with this error:\r\n\r\n```\r\n2015-11-30 03:35:13 +0000 [warn]: record_reformer: Fluent::BufferQueueLimitError queue size exceeds limit /opt/google-fluentd/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/buffer.rb:189:in `block in emit'\r\n2015-11-30 03:35:14 +0000 [warn]: emit transaction failed: error_class=Fluent::BufferQueueLimitError error=\"queue size exceeds limit\" tag=\"kubernetes.fluentd-cloud-logging-k8s-yasp-minion-szrb_kube-system_fluentd-cloud-logging\"\r\n  2015-11-30 03:35:14 +0000 [warn]: suppressed same stacktrace\r\n```\r\n\r\nAny fix?"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/160747297","html_url":"https://github.com/kubernetes/kubernetes/issues/17929#issuecomment-160747297","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17929","id":160747297,"user":{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},"created_at":"2015-11-30T20:14:41Z","updated_at":"2015-11-30T20:14:41Z","body":"@paralin can you run the command: kubectl describe pods fluentd-cloud-logging-k8s-yasp-minion-p5pm  and copy&paste the output here? "}}