{"action":"created","issue":{"url":"https://api.github.com/repos/DataDog/dd-agent/issues/1577","labels_url":"https://api.github.com/repos/DataDog/dd-agent/issues/1577/labels{/name}","comments_url":"https://api.github.com/repos/DataDog/dd-agent/issues/1577/comments","events_url":"https://api.github.com/repos/DataDog/dd-agent/issues/1577/events","html_url":"https://github.com/DataDog/dd-agent/pull/1577","id":70727649,"number":1577,"title":"Talwai/agent dev mode","user":{"login":"talwai","id":3237580,"avatar_url":"https://avatars.githubusercontent.com/u/3237580?v=3","gravatar_id":"","url":"https://api.github.com/users/talwai","html_url":"https://github.com/talwai","followers_url":"https://api.github.com/users/talwai/followers","following_url":"https://api.github.com/users/talwai/following{/other_user}","gists_url":"https://api.github.com/users/talwai/gists{/gist_id}","starred_url":"https://api.github.com/users/talwai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talwai/subscriptions","organizations_url":"https://api.github.com/users/talwai/orgs","repos_url":"https://api.github.com/users/talwai/repos","events_url":"https://api.github.com/users/talwai/events{/privacy}","received_events_url":"https://api.github.com/users/talwai/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-04-24T16:03:17Z","updated_at":"2015-04-24T21:06:20Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/DataDog/dd-agent/pulls/1577","html_url":"https://github.com/DataDog/dd-agent/pull/1577","diff_url":"https://github.com/DataDog/dd-agent/pull/1577.diff","patch_url":"https://github.com/DataDog/dd-agent/pull/1577.patch"},"body":"This adds a \"developer mode\" to the agent, that can be enabled by the `--profile` command line flag or by the `developer_mode` setting in `agentConfig`. \r\n\r\nDeveloper mode can be enabled at the `check` level or the `collector` level. When enabled at the check\r\nlevel, e.g. `./agent.py check nginx --profile`, the `check.run()` function is profiled with `cProfile` and the `pstats` output is dumped to the log. \r\n\r\nWhen enabled at the `collector` level, the following behavior is enabled:\r\n- Every run of the collector loop is profiled with `cProfile`, in the same way as above\r\n- The `AgentMetrics` check (`checks.d/agent_metrics.py`) is run at the end of each collector loop. This check can be configured to run a set of additional `psutil.Process` methods on the current process. These additional metrics are flushed to the dispatcher as well as dumped to the log. See `conf.d/agent_metrics.yaml.default` for the default configuration, which can be extended as needed. Currently a setting under `process_metrics` is ignored if there is no corresponding `psutil.Process` method.\r\n"},"comment":{"url":"https://api.github.com/repos/DataDog/dd-agent/issues/comments/96065384","html_url":"https://github.com/DataDog/dd-agent/pull/1577#issuecomment-96065384","issue_url":"https://api.github.com/repos/DataDog/dd-agent/issues/1577","id":96065384,"user":{"login":"talwai","id":3237580,"avatar_url":"https://avatars.githubusercontent.com/u/3237580?v=3","gravatar_id":"","url":"https://api.github.com/users/talwai","html_url":"https://github.com/talwai","followers_url":"https://api.github.com/users/talwai/followers","following_url":"https://api.github.com/users/talwai/following{/other_user}","gists_url":"https://api.github.com/users/talwai/gists{/gist_id}","starred_url":"https://api.github.com/users/talwai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talwai/subscriptions","organizations_url":"https://api.github.com/users/talwai/orgs","repos_url":"https://api.github.com/users/talwai/repos","events_url":"https://api.github.com/users/talwai/events{/privacy}","received_events_url":"https://api.github.com/users/talwai/received_events","type":"User","site_admin":false},"created_at":"2015-04-24T21:06:20Z","updated_at":"2015-04-24T21:06:20Z","body":"@degemer Thanks for the feedback!\r\n> I was expecting a lot more \"by check\" metrics, ie at the end of every run, display some stats for each check (and not only globally), for instance run time, memory consumption after/before, io read/write \r\n\r\nI agree that we should be getting more data about the performance of individual checks, and not just the collector run as a whole. I can extend `wrap_profiling` to include some before and after `psutil` calls. Though I wonder if the extra system calls would hinder the performance of the check itself i.e. `check.run()` becomes `psutil.get_memory_info() ; check.run() ; psutil.get_memory_info()` for every check. \r\n> Also, about these metrics, could you display them in a nicer way ? \r\n\r\nYep, the output is ugly. Will improve the formatting for the log dumps\r\n\r\n> I think that the profiler should be another option, not activated by default with the \"developer mode\", because we don't always need this level of details when debugging a check. And maybe also a profiler by check ? (not sure it's needed, just a suggestion)\r\n\r\nI get where you're coming from. In fact, I myself silenced the profiler dump when debugging the `AgentMetrics` check because it was mostly unhelpful. I wonder if that is the case in general though. Easy enough to make them two separate options, but would be interested in hearing other opinions.\r\n\r\n> Another thing: I don't think you should put agent_metrics.py in checks.d, because when you do this it becomes visible in\r\ninitialized checks.d checks: ['agent_metrics', 'network', 'pgbouncer', 'ntp', 'consul', 'nginx']\r\n\r\nThe reason it's in there is: \r\n1. Because it appears that that's where all the new-style checks go \r\n2. So that it can be tested via `AgentCheckTest`. \r\nI realize that it is possible to test without the scaffolding, but my thought was that all tests for new-style checks should follow the same pattern. If the only issue with locating it in `checks.d/` is the log output, then I would prefer to just special-case it so it doesn't show up. Admittedly there may be other issues with having it in `checks.d/` that I'm not aware of.\r\n\r\n> I just realized that we already have some agent_metrics here: https://github.com/DataDog/dd-agent/blob/master/checks/agent_metrics.py, you might want to put yours there too. (or maybe you split them on purpose ?)\r\n\r\nBasically `checks.d/agent_metrics.py` is all of `checks/agent_metrics.py` and more. I opted against simply extending the class in place because it uses the old `Check` API, and if we plan to deprecate that API we should probably stop using it internally."}}