{"action":"created","issue":{"url":"https://api.github.com/repos/DataDog/dd-agent/issues/1892","labels_url":"https://api.github.com/repos/DataDog/dd-agent/issues/1892/labels{/name}","comments_url":"https://api.github.com/repos/DataDog/dd-agent/issues/1892/comments","events_url":"https://api.github.com/repos/DataDog/dd-agent/issues/1892/events","html_url":"https://github.com/DataDog/dd-agent/pull/1892","id":104787897,"number":1892,"title":"Replace subprocess.Popen with safer get_subprocess_output","user":{"login":"JohnLZeller","id":1171118,"avatar_url":"https://avatars.githubusercontent.com/u/1171118?v=3","gravatar_id":"","url":"https://api.github.com/users/JohnLZeller","html_url":"https://github.com/JohnLZeller","followers_url":"https://api.github.com/users/JohnLZeller/followers","following_url":"https://api.github.com/users/JohnLZeller/following{/other_user}","gists_url":"https://api.github.com/users/JohnLZeller/gists{/gist_id}","starred_url":"https://api.github.com/users/JohnLZeller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JohnLZeller/subscriptions","organizations_url":"https://api.github.com/users/JohnLZeller/orgs","repos_url":"https://api.github.com/users/JohnLZeller/repos","events_url":"https://api.github.com/users/JohnLZeller/events{/privacy}","received_events_url":"https://api.github.com/users/JohnLZeller/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/DataDog/dd-agent/milestones/55","html_url":"https://github.com/DataDog/dd-agent/milestones/5.5.2","labels_url":"https://api.github.com/repos/DataDog/dd-agent/milestones/55/labels","id":1363951,"number":55,"title":"5.5.2","description":null,"creator":{"login":"remh","id":1311477,"avatar_url":"https://avatars.githubusercontent.com/u/1311477?v=3","gravatar_id":"","url":"https://api.github.com/users/remh","html_url":"https://github.com/remh","followers_url":"https://api.github.com/users/remh/followers","following_url":"https://api.github.com/users/remh/following{/other_user}","gists_url":"https://api.github.com/users/remh/gists{/gist_id}","starred_url":"https://api.github.com/users/remh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/remh/subscriptions","organizations_url":"https://api.github.com/users/remh/orgs","repos_url":"https://api.github.com/users/remh/repos","events_url":"https://api.github.com/users/remh/events{/privacy}","received_events_url":"https://api.github.com/users/remh/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":1,"state":"open","created_at":"2015-10-19T13:46:26Z","updated_at":"2015-10-21T20:51:46Z","due_on":null,"closed_at":null},"comments":12,"created_at":"2015-09-03T21:46:10Z","updated_at":"2015-10-21T20:57:06Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/DataDog/dd-agent/pulls/1892","html_url":"https://github.com/DataDog/dd-agent/pull/1892","diff_url":"https://github.com/DataDog/dd-agent/pull/1892.diff","patch_url":"https://github.com/DataDog/dd-agent/pull/1892.patch"},"body":"Issue came up when investigating hostdown issues.\r\nhttps://datadog.zendesk.com/agent/tickets/30510\r\n\r\nIt seems as though, periodically, a subprocess call hangs and ultimately causes the watchdog to reset the agent. During this period of time, there is then a host down event, followed by a host recovered event posted to the customers events page. We noticed that it seems as though the hanging happens in various checks (not the same for every host down event). Offending checks have included the customers own custom `iostat` check, as well as our own `postfix` check, and even `resources/processes.py`.\r\n\r\nThe theory is that this is caused by a hanging call to `subprocess.Popen`. There is a known issue where the use of a `subprocess.PIPE` for stdout (or even stderr) could cause the process to hang once the `subprocess.PIPE` fills up. The solution to this is to use `get_subprocess_output()` in our own `utils/subprocess_output.py`, which uses a file for stdout and stderr, removing such a low memory capacity issue.\r\n\r\nTo avoid further chasing this problem around the agent, this PR attempts to replace the use of `subprocess.Popen` anywhere it shows up in the agent with `get_subprocess_output()`. There are a few instances where further work will be needed to refactor the code in a clean way, and for those instances, I have added a `FIXME` note."},"comment":{"url":"https://api.github.com/repos/DataDog/dd-agent/issues/comments/150022045","html_url":"https://github.com/DataDog/dd-agent/pull/1892#issuecomment-150022045","issue_url":"https://api.github.com/repos/DataDog/dd-agent/issues/1892","id":150022045,"user":{"login":"JohnLZeller","id":1171118,"avatar_url":"https://avatars.githubusercontent.com/u/1171118?v=3","gravatar_id":"","url":"https://api.github.com/users/JohnLZeller","html_url":"https://github.com/JohnLZeller","followers_url":"https://api.github.com/users/JohnLZeller/followers","following_url":"https://api.github.com/users/JohnLZeller/following{/other_user}","gists_url":"https://api.github.com/users/JohnLZeller/gists{/gist_id}","starred_url":"https://api.github.com/users/JohnLZeller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JohnLZeller/subscriptions","organizations_url":"https://api.github.com/users/JohnLZeller/orgs","repos_url":"https://api.github.com/users/JohnLZeller/repos","events_url":"https://api.github.com/users/JohnLZeller/events{/privacy}","received_events_url":"https://api.github.com/users/JohnLZeller/received_events","type":"User","site_admin":false},"created_at":"2015-10-21T20:57:06Z","updated_at":"2015-10-21T20:57:06Z","body":"@remh will do"}}