{"action":"created","comment":{"url":"https://api.github.com/repos/DataDog/dd-agent/pulls/comments/22940541","id":22940541,"diff_hunk":"@@ -0,0 +1,69 @@\n+Testing & dd-agent\n+==================\n+\n+[![Build Status](https://travis-ci.org/DataDog/dd-agent.svg)](https://travis-ci.org/DataDog/dd-agent)\n+\n+# Lint\n+\n+Your code should always be clean when doing `rake lint`.\n+The pylint configuration is not really aggressive so you're most welcome to use your own pylint config or tools like pep8 or pyflakes (we plan to plug in more restrictive stuff about that to try to keep sane defaults in the codebase)\n+\n+# Unit tests\n+\n+They mainly concern the core of the Datadog agent:\n+\n+* how metrics are handled/submitted/aggregated\n+* how we run checks, handle failures\n+* more generic testing on the agent behavior and resilience\n+* system-oriented checks (may be considered as integration tests..)\n+\n+```\n+# Run the suite\n+rake ci:run\n+```\n+\n+# Integration tests\n+\n+They ensure that the agent is correctly talking to third party software which is necessary for most checks.\n+\n+They are great because they mimic a real setup where someone would enable a check on a machine with this service running. Using mocks or pre-saved responses often hides corner-cases and are the source of lots of issues.\n+\n+We run these tests by creating a build machine \"flavor\" (see Travis section), basically each flavor is defined by the third party software we install on this machine.\n+\n+Most of the times `flavor == check_name`\n+\n+Each flavor is defined in `ci/flavor.rb`, and we set different steps for running the flavor build :\n+\n+* **before_install** needs to be run before installation\n+* **install** installs the 3p software\n+* **before_script** generally setups the software and launch it in background\n+* **script** runs nosetests with a filter (see how to write tests)\n+* **cleanup** stops the software and remove unnecessary data (not running on Travis, because the buildboxes are disposable)\n+\n+Your test cases must be written in `tests/test_flavor.rb` and they must use the nose `attr` decorator to be filtered by the flavors.","path":"tests/README.md","position":43,"original_position":43,"commit_id":"31a99c7be03c6499bfb834df4dd55434007149dc","original_commit_id":"31a99c7be03c6499bfb834df4dd55434007149dc","user":{"login":"remh","id":1311477,"avatar_url":"https://avatars.githubusercontent.com/u/1311477?v=3","gravatar_id":"","url":"https://api.github.com/users/remh","html_url":"https://github.com/remh","followers_url":"https://api.github.com/users/remh/followers","following_url":"https://api.github.com/users/remh/following{/other_user}","gists_url":"https://api.github.com/users/remh/gists{/gist_id}","starred_url":"https://api.github.com/users/remh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/remh/subscriptions","organizations_url":"https://api.github.com/users/remh/orgs","repos_url":"https://api.github.com/users/remh/repos","events_url":"https://api.github.com/users/remh/events{/privacy}","received_events_url":"https://api.github.com/users/remh/received_events","type":"User","site_admin":false},"body":"```\r\ntests/test_flavor.py\r\n```","created_at":"2015-01-14T14:51:30Z","updated_at":"2015-01-14T14:51:30Z","html_url":"https://github.com/DataDog/dd-agent/pull/1293#discussion_r22940541","pull_request_url":"https://api.github.com/repos/DataDog/dd-agent/pulls/1293","_links":{"self":{"href":"https://api.github.com/repos/DataDog/dd-agent/pulls/comments/22940541"},"html":{"href":"https://github.com/DataDog/dd-agent/pull/1293#discussion_r22940541"},"pull_request":{"href":"https://api.github.com/repos/DataDog/dd-agent/pulls/1293"}}},"pull_request":{"url":"https://api.github.com/repos/DataDog/dd-agent/pulls/1293","id":27321002,"html_url":"https://github.com/DataDog/dd-agent/pull/1293","diff_url":"https://github.com/DataDog/dd-agent/pull/1293.diff","patch_url":"https://github.com/DataDog/dd-agent/pull/1293.patch","issue_url":"https://api.github.com/repos/DataDog/dd-agent/issues/1293","number":1293,"state":"open","locked":false,"title":"[ci/tests] New build. New Infrastructure. New Flavors","user":{"login":"LeoCavaille","id":1788830,"avatar_url":"https://avatars.githubusercontent.com/u/1788830?v=3","gravatar_id":"","url":"https://api.github.com/users/LeoCavaille","html_url":"https://github.com/LeoCavaille","followers_url":"https://api.github.com/users/LeoCavaille/followers","following_url":"https://api.github.com/users/LeoCavaille/following{/other_user}","gists_url":"https://api.github.com/users/LeoCavaille/gists{/gist_id}","starred_url":"https://api.github.com/users/LeoCavaille/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LeoCavaille/subscriptions","organizations_url":"https://api.github.com/users/LeoCavaille/orgs","repos_url":"https://api.github.com/users/LeoCavaille/repos","events_url":"https://api.github.com/users/LeoCavaille/events{/privacy}","received_events_url":"https://api.github.com/users/LeoCavaille/received_events","type":"User","site_admin":false},"body":"Much more testing.\r\nSuch multiple versions.\r\nVery docker-blazing-speed\r\nSo hip new PG tests\r\n:dog: \r\n\r\nThe first build will probably a bit slow, subsequent builds should be faster thanks to pre-compiled software caches.\r\nAnd more seriously see commit messages:\r\n\r\nThe builds must not rely on sudo from now on.\r\nThat means either fetching pre-compiled binaries from\r\nupstream or fetching the sources and compiling directly\r\non the build machine.\r\nHowever, as the released upstream sources do not change\r\nwe can benefit from the caching system and cache the\r\nbinaries once we have them.\r\nAll these software are in $INTEGRATIONS_DIR\r\n\r\nIntroduction also of running the integration tests against\r\ndifferent versions of the software described in the build\r\nmatrix.\r\n\r\nAlso a bit of documentation on how to use the new test system.\r\n\r\n+ Several more fixes of the test suite codebase","created_at":"2015-01-14T01:41:40Z","updated_at":"2015-01-14T14:51:30Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bc4b8d459479ad2888791f0443658fc2b8ea17d7","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/DataDog/dd-agent/pulls/1293/commits","review_comments_url":"https://api.github.com/repos/DataDog/dd-agent/pulls/1293/comments","review_comment_url":"https://api.github.com/repos/DataDog/dd-agent/pulls/comments/{number}","comments_url":"https://api.github.com/repos/DataDog/dd-agent/issues/1293/comments","statuses_url":"https://api.github.com/repos/DataDog/dd-agent/statuses/31a99c7be03c6499bfb834df4dd55434007149dc","head":{"label":"DataDog:leo/docker_travis","ref":"leo/docker_travis","sha":"31a99c7be03c6499bfb834df4dd55434007149dc","user":{"login":"DataDog","id":365230,"avatar_url":"https://avatars.githubusercontent.com/u/365230?v=3","gravatar_id":"","url":"https://api.github.com/users/DataDog","html_url":"https://github.com/DataDog","followers_url":"https://api.github.com/users/DataDog/followers","following_url":"https://api.github.com/users/DataDog/following{/other_user}","gists_url":"https://api.github.com/users/DataDog/gists{/gist_id}","starred_url":"https://api.github.com/users/DataDog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DataDog/subscriptions","organizations_url":"https://api.github.com/users/DataDog/orgs","repos_url":"https://api.github.com/users/DataDog/repos","events_url":"https://api.github.com/users/DataDog/events{/privacy}","received_events_url":"https://api.github.com/users/DataDog/received_events","type":"Organization","site_admin":false},"repo":{"id":1210071,"name":"dd-agent","full_name":"DataDog/dd-agent","owner":{"login":"DataDog","id":365230,"avatar_url":"https://avatars.githubusercontent.com/u/365230?v=3","gravatar_id":"","url":"https://api.github.com/users/DataDog","html_url":"https://github.com/DataDog","followers_url":"https://api.github.com/users/DataDog/followers","following_url":"https://api.github.com/users/DataDog/following{/other_user}","gists_url":"https://api.github.com/users/DataDog/gists{/gist_id}","starred_url":"https://api.github.com/users/DataDog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DataDog/subscriptions","organizations_url":"https://api.github.com/users/DataDog/orgs","repos_url":"https://api.github.com/users/DataDog/repos","events_url":"https://api.github.com/users/DataDog/events{/privacy}","received_events_url":"https://api.github.com/users/DataDog/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/DataDog/dd-agent","description":"Datadog Agent","fork":false,"url":"https://api.github.com/repos/DataDog/dd-agent","forks_url":"https://api.github.com/repos/DataDog/dd-agent/forks","keys_url":"https://api.github.com/repos/DataDog/dd-agent/keys{/key_id}","collaborators_url":"https://api.github.com/repos/DataDog/dd-agent/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/DataDog/dd-agent/teams","hooks_url":"https://api.github.com/repos/DataDog/dd-agent/hooks","issue_events_url":"https://api.github.com/repos/DataDog/dd-agent/issues/events{/number}","events_url":"https://api.github.com/repos/DataDog/dd-agent/events","assignees_url":"https://api.github.com/repos/DataDog/dd-agent/assignees{/user}","branches_url":"https://api.github.com/repos/DataDog/dd-agent/branches{/branch}","tags_url":"https://api.github.com/repos/DataDog/dd-agent/tags","blobs_url":"https://api.github.com/repos/DataDog/dd-agent/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/DataDog/dd-agent/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/DataDog/dd-agent/git/refs{/sha}","trees_url":"https://api.github.com/repos/DataDog/dd-agent/git/trees{/sha}","statuses_url":"https://api.github.com/repos/DataDog/dd-agent/statuses/{sha}","languages_url":"https://api.github.com/repos/DataDog/dd-agent/languages","stargazers_url":"https://api.github.com/repos/DataDog/dd-agent/stargazers","contributors_url":"https://api.github.com/repos/DataDog/dd-agent/contributors","subscribers_url":"https://api.github.com/repos/DataDog/dd-agent/subscribers","subscription_url":"https://api.github.com/repos/DataDog/dd-agent/subscription","commits_url":"https://api.github.com/repos/DataDog/dd-agent/commits{/sha}","git_commits_url":"https://api.github.com/repos/DataDog/dd-agent/git/commits{/sha}","comments_url":"https://api.github.com/repos/DataDog/dd-agent/comments{/number}","issue_comment_url":"https://api.github.com/repos/DataDog/dd-agent/issues/comments/{number}","contents_url":"https://api.github.com/repos/DataDog/dd-agent/contents/{+path}","compare_url":"https://api.github.com/repos/DataDog/dd-agent/compare/{base}...{head}","merges_url":"https://api.github.com/repos/DataDog/dd-agent/merges","archive_url":"https://api.github.com/repos/DataDog/dd-agent/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/DataDog/dd-agent/downloads","issues_url":"https://api.github.com/repos/DataDog/dd-agent/issues{/number}","pulls_url":"https://api.github.com/repos/DataDog/dd-agent/pulls{/number}","milestones_url":"https://api.github.com/repos/DataDog/dd-agent/milestones{/number}","notifications_url":"https://api.github.com/repos/DataDog/dd-agent/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/DataDog/dd-agent/labels{/name}","releases_url":"https://api.github.com/repos/DataDog/dd-agent/releases{/id}","created_at":"2010-12-31T03:02:47Z","updated_at":"2015-01-13T20:45:58Z","pushed_at":"2015-01-14T14:45:36Z","git_url":"git://github.com/DataDog/dd-agent.git","ssh_url":"git@github.com:DataDog/dd-agent.git","clone_url":"https://github.com/DataDog/dd-agent.git","svn_url":"https://github.com/DataDog/dd-agent","homepage":"http://docs.datadoghq.com/","size":41292,"stargazers_count":239,"watchers_count":239,"language":"Python","has_issues":true,"has_downloads":false,"has_wiki":true,"has_pages":true,"forks_count":153,"mirror_url":null,"open_issues_count":170,"forks":153,"open_issues":170,"watchers":239,"default_branch":"master"}},"base":{"label":"DataDog:master","ref":"master","sha":"313b21edcbb1e57b042c1bea6a1e816b77b0659f","user":{"login":"DataDog","id":365230,"avatar_url":"https://avatars.githubusercontent.com/u/365230?v=3","gravatar_id":"","url":"https://api.github.com/users/DataDog","html_url":"https://github.com/DataDog","followers_url":"https://api.github.com/users/DataDog/followers","following_url":"https://api.github.com/users/DataDog/following{/other_user}","gists_url":"https://api.github.com/users/DataDog/gists{/gist_id}","starred_url":"https://api.github.com/users/DataDog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DataDog/subscriptions","organizations_url":"https://api.github.com/users/DataDog/orgs","repos_url":"https://api.github.com/users/DataDog/repos","events_url":"https://api.github.com/users/DataDog/events{/privacy}","received_events_url":"https://api.github.com/users/DataDog/received_events","type":"Organization","site_admin":false},"repo":{"id":1210071,"name":"dd-agent","full_name":"DataDog/dd-agent","owner":{"login":"DataDog","id":365230,"avatar_url":"https://avatars.githubusercontent.com/u/365230?v=3","gravatar_id":"","url":"https://api.github.com/users/DataDog","html_url":"https://github.com/DataDog","followers_url":"https://api.github.com/users/DataDog/followers","following_url":"https://api.github.com/users/DataDog/following{/other_user}","gists_url":"https://api.github.com/users/DataDog/gists{/gist_id}","starred_url":"https://api.github.com/users/DataDog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DataDog/subscriptions","organizations_url":"https://api.github.com/users/DataDog/orgs","repos_url":"https://api.github.com/users/DataDog/repos","events_url":"https://api.github.com/users/DataDog/events{/privacy}","received_events_url":"https://api.github.com/users/DataDog/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/DataDog/dd-agent","description":"Datadog Agent","fork":false,"url":"https://api.github.com/repos/DataDog/dd-agent","forks_url":"https://api.github.com/repos/DataDog/dd-agent/forks","keys_url":"https://api.github.com/repos/DataDog/dd-agent/keys{/key_id}","collaborators_url":"https://api.github.com/repos/DataDog/dd-agent/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/DataDog/dd-agent/teams","hooks_url":"https://api.github.com/repos/DataDog/dd-agent/hooks","issue_events_url":"https://api.github.com/repos/DataDog/dd-agent/issues/events{/number}","events_url":"https://api.github.com/repos/DataDog/dd-agent/events","assignees_url":"https://api.github.com/repos/DataDog/dd-agent/assignees{/user}","branches_url":"https://api.github.com/repos/DataDog/dd-agent/branches{/branch}","tags_url":"https://api.github.com/repos/DataDog/dd-agent/tags","blobs_url":"https://api.github.com/repos/DataDog/dd-agent/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/DataDog/dd-agent/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/DataDog/dd-agent/git/refs{/sha}","trees_url":"https://api.github.com/repos/DataDog/dd-agent/git/trees{/sha}","statuses_url":"https://api.github.com/repos/DataDog/dd-agent/statuses/{sha}","languages_url":"https://api.github.com/repos/DataDog/dd-agent/languages","stargazers_url":"https://api.github.com/repos/DataDog/dd-agent/stargazers","contributors_url":"https://api.github.com/repos/DataDog/dd-agent/contributors","subscribers_url":"https://api.github.com/repos/DataDog/dd-agent/subscribers","subscription_url":"https://api.github.com/repos/DataDog/dd-agent/subscription","commits_url":"https://api.github.com/repos/DataDog/dd-agent/commits{/sha}","git_commits_url":"https://api.github.com/repos/DataDog/dd-agent/git/commits{/sha}","comments_url":"https://api.github.com/repos/DataDog/dd-agent/comments{/number}","issue_comment_url":"https://api.github.com/repos/DataDog/dd-agent/issues/comments/{number}","contents_url":"https://api.github.com/repos/DataDog/dd-agent/contents/{+path}","compare_url":"https://api.github.com/repos/DataDog/dd-agent/compare/{base}...{head}","merges_url":"https://api.github.com/repos/DataDog/dd-agent/merges","archive_url":"https://api.github.com/repos/DataDog/dd-agent/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/DataDog/dd-agent/downloads","issues_url":"https://api.github.com/repos/DataDog/dd-agent/issues{/number}","pulls_url":"https://api.github.com/repos/DataDog/dd-agent/pulls{/number}","milestones_url":"https://api.github.com/repos/DataDog/dd-agent/milestones{/number}","notifications_url":"https://api.github.com/repos/DataDog/dd-agent/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/DataDog/dd-agent/labels{/name}","releases_url":"https://api.github.com/repos/DataDog/dd-agent/releases{/id}","created_at":"2010-12-31T03:02:47Z","updated_at":"2015-01-13T20:45:58Z","pushed_at":"2015-01-14T14:45:36Z","git_url":"git://github.com/DataDog/dd-agent.git","ssh_url":"git@github.com:DataDog/dd-agent.git","clone_url":"https://github.com/DataDog/dd-agent.git","svn_url":"https://github.com/DataDog/dd-agent","homepage":"http://docs.datadoghq.com/","size":41292,"stargazers_count":239,"watchers_count":239,"language":"Python","has_issues":true,"has_downloads":false,"has_wiki":true,"has_pages":true,"forks_count":153,"mirror_url":null,"open_issues_count":170,"forks":153,"open_issues":170,"watchers":239,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/DataDog/dd-agent/pulls/1293"},"html":{"href":"https://github.com/DataDog/dd-agent/pull/1293"},"issue":{"href":"https://api.github.com/repos/DataDog/dd-agent/issues/1293"},"comments":{"href":"https://api.github.com/repos/DataDog/dd-agent/issues/1293/comments"},"review_comments":{"href":"https://api.github.com/repos/DataDog/dd-agent/pulls/1293/comments"},"review_comment":{"href":"https://api.github.com/repos/DataDog/dd-agent/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/DataDog/dd-agent/pulls/1293/commits"},"statuses":{"href":"https://api.github.com/repos/DataDog/dd-agent/statuses/31a99c7be03c6499bfb834df4dd55434007149dc"}}}}