{"action":"created","comment":{"url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/comments/44674441","id":44674441,"diff_hunk":"@@ -0,0 +1,71 @@\n+# Copyright ClusterHQ Inc.  See LICENSE file for details.\n+\n+\"\"\"\n+Base classes for unit tests.\n+\"\"\"\n+\n+import fixtures\n+import testtools\n+from testtools.deferredruntest import (\n+    AsynchronousDeferredRunTestForBrokenTwisted)\n+\n+from twisted.python.filepath import FilePath\n+from twisted.trial import unittest\n+\n+\n+class TestCase(unittest.SynchronousTestCase):\n+    \"\"\"\n+    Base class for synchronous test cases.\n+    \"\"\"\n+\n+\n+def async_runner(timeout):\n+    \"\"\"\n+    Make a ``RunTest`` instance for asynchronous tests.\n+\n+    :param float timeout: The maximum length of time (in seconds) that a test\n+        is allowed to take.\n+    \"\"\"\n+    # XXX: Looks like the acceptance tests (which were the first tests that we\n+    # tried to migrate) aren't cleaning up after themselves even in the\n+    # successful case. Use the RunTest that loops the reactor a couple of\n+    # times after the test is done.\n+    return AsynchronousDeferredRunTestForBrokenTwisted.make_factory(\n+        timeout=timeout)\n+\n+\n+# By default, asynchronous tests are timed out after 2 minutes.\n+DEFAULT_ASYNC_TIMEOUT = 120\n+\n+\n+def _test_skipped(case, result, exception):\n+    result.addSkip(case, str(exception))\n+\n+\n+class AsyncTestCase(testtools.TestCase):\n+    \"\"\"\n+    Base class for asynchronous test cases.\n+    \"\"\"\n+\n+    run_tests_with = async_runner(timeout=DEFAULT_ASYNC_TIMEOUT)\n+\n+    def __init__(self, *args, **kwargs):\n+        super(AsyncTestCase, self).__init__(*args, **kwargs)\n+        self.exception_handlers.insert(-1, (unittest.SkipTest, _test_skipped))","path":"flocker/testtools/_base.py","position":54,"original_position":54,"commit_id":"325ec0cfd2dc8a70ff9b64b08576c5e88a413f32","original_commit_id":"325ec0cfd2dc8a70ff9b64b08576c5e88a413f32","user":{"login":"jml","id":125674,"avatar_url":"https://avatars.githubusercontent.com/u/125674?v=3","gravatar_id":"","url":"https://api.github.com/users/jml","html_url":"https://github.com/jml","followers_url":"https://api.github.com/users/jml/followers","following_url":"https://api.github.com/users/jml/following{/other_user}","gists_url":"https://api.github.com/users/jml/gists{/gist_id}","starred_url":"https://api.github.com/users/jml/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jml/subscriptions","organizations_url":"https://api.github.com/users/jml/orgs","repos_url":"https://api.github.com/users/jml/repos","events_url":"https://api.github.com/users/jml/events{/privacy}","received_events_url":"https://api.github.com/users/jml/received_events","type":"User","site_admin":false},"body":"Yeah, sorry. At least this wart is documented. ","created_at":"2015-11-12T15:59:02Z","updated_at":"2015-11-12T15:59:02Z","html_url":"https://github.com/ClusterHQ/flocker/pull/2163#discussion_r44674441","pull_request_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2163","_links":{"self":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/comments/44674441"},"html":{"href":"https://github.com/ClusterHQ/flocker/pull/2163#discussion_r44674441"},"pull_request":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2163"}}},"pull_request":{"url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2163","id":50423809,"html_url":"https://github.com/ClusterHQ/flocker/pull/2163","diff_url":"https://github.com/ClusterHQ/flocker/pull/2163.diff","patch_url":"https://github.com/ClusterHQ/flocker/pull/2163.patch","issue_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2163","number":2163,"state":"open","locked":false,"title":"[FLOC-3412] New base TestCase ","user":{"login":"jml","id":125674,"avatar_url":"https://avatars.githubusercontent.com/u/125674?v=3","gravatar_id":"","url":"https://api.github.com/users/jml","html_url":"https://github.com/jml","followers_url":"https://api.github.com/users/jml/followers","following_url":"https://api.github.com/users/jml/following{/other_user}","gists_url":"https://api.github.com/users/jml/gists{/gist_id}","starred_url":"https://api.github.com/users/jml/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jml/subscriptions","organizations_url":"https://api.github.com/users/jml/orgs","repos_url":"https://api.github.com/users/jml/repos","events_url":"https://api.github.com/users/jml/events{/privacy}","received_events_url":"https://api.github.com/users/jml/received_events","type":"User","site_admin":false},"body":"In order to implement retries for flaky tests, we're going to have to customize the way tests are run, because we want to retry `setUp`, the test method itself, and `tearDown`.\r\n\r\n[testtools](http://testtools.readthedocs.org/en/latest/) has a built-in mechanism for customizing the behaviour of how a test is run without retrying (using the [RunTest abstraction](http://testtools.readthedocs.org/en/latest/for-framework-folk.html#controlling-test-execution)). Because these are composable, we can combine our (future, not yet implemented) `RetryFlakyTest` runner with the runners that testtools provides for Twisted tests. \r\n\r\nThe implementation strategy here is to create two new \"official\" base classes for flocker tests: `flocker.testtools.TestCase` and `flocker.testtools.AsyncTestCase`. In later branches, all existing tests will be migrated to use these base cases. For now, I've focused only on the acceptance tests, because they are the most flaky ones.\r\n\r\nA later branch will add a runner for retrying flaky tests. \r\n\r\nIn an attempt to actually _add_ value, I've done a little bit of extra work to provide pretty logs with test failures. Unfortunately, I haven't yet got a good example of how it looks. \r\n\r\nSome context:\r\n\r\ntesttools has a feature for attaching arbitrary stuff to a test and having that stuff show up when the test fails. For Twisted tests, it attaches the entire log that occurred while the test was run, e.g.\r\nhttp://build.clusterhq.com/builders/flocker%2Facceptance%2Frackspace%2Fubuntu-14.04%2Frackspace/builds/3324/steps/trial/logs/problems\r\n\r\nBecause we log a lot of stuff to Twisted in our acceptance tests, I thought it would be nice to separate out the eliot logs & pretty print them. This will hopefully make failures easier to debug. When I get an example I'll link it here.\r\n\r\nAs usual, I've added `XXX` comments where I'd particularly like feedback about a trade-off I made or a decision that I'm not entirely comfortable about.\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/clusterhq/flocker/2163)\n<!-- Reviewable:end -->\n","created_at":"2015-11-11T19:19:15Z","updated_at":"2015-11-12T15:59:02Z","closed_at":null,"merged_at":null,"merge_commit_sha":"55b5bd19a82ca6f4634f7c34d71cd8686aa18571","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2163/commits","review_comments_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2163/comments","review_comment_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls/comments{/number}","comments_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/2163/comments","statuses_url":"https://api.github.com/repos/ClusterHQ/flocker/statuses/325ec0cfd2dc8a70ff9b64b08576c5e88a413f32","head":{"label":"ClusterHQ:new-base-FLOC-3412","ref":"new-base-FLOC-3412","sha":"325ec0cfd2dc8a70ff9b64b08576c5e88a413f32","user":{"login":"ClusterHQ","id":7913449,"avatar_url":"https://avatars.githubusercontent.com/u/7913449?v=3","gravatar_id":"","url":"https://api.github.com/users/ClusterHQ","html_url":"https://github.com/ClusterHQ","followers_url":"https://api.github.com/users/ClusterHQ/followers","following_url":"https://api.github.com/users/ClusterHQ/following{/other_user}","gists_url":"https://api.github.com/users/ClusterHQ/gists{/gist_id}","starred_url":"https://api.github.com/users/ClusterHQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClusterHQ/subscriptions","organizations_url":"https://api.github.com/users/ClusterHQ/orgs","repos_url":"https://api.github.com/users/ClusterHQ/repos","events_url":"https://api.github.com/users/ClusterHQ/events{/privacy}","received_events_url":"https://api.github.com/users/ClusterHQ/received_events","type":"Organization","site_admin":false},"repo":{"id":19248542,"name":"flocker","full_name":"ClusterHQ/flocker","owner":{"login":"ClusterHQ","id":7913449,"avatar_url":"https://avatars.githubusercontent.com/u/7913449?v=3","gravatar_id":"","url":"https://api.github.com/users/ClusterHQ","html_url":"https://github.com/ClusterHQ","followers_url":"https://api.github.com/users/ClusterHQ/followers","following_url":"https://api.github.com/users/ClusterHQ/following{/other_user}","gists_url":"https://api.github.com/users/ClusterHQ/gists{/gist_id}","starred_url":"https://api.github.com/users/ClusterHQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClusterHQ/subscriptions","organizations_url":"https://api.github.com/users/ClusterHQ/orgs","repos_url":"https://api.github.com/users/ClusterHQ/repos","events_url":"https://api.github.com/users/ClusterHQ/events{/privacy}","received_events_url":"https://api.github.com/users/ClusterHQ/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ClusterHQ/flocker","description":"Container data volume manager for your Dockerized application","fork":false,"url":"https://api.github.com/repos/ClusterHQ/flocker","forks_url":"https://api.github.com/repos/ClusterHQ/flocker/forks","keys_url":"https://api.github.com/repos/ClusterHQ/flocker/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ClusterHQ/flocker/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ClusterHQ/flocker/teams","hooks_url":"https://api.github.com/repos/ClusterHQ/flocker/hooks","issue_events_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/events{/number}","events_url":"https://api.github.com/repos/ClusterHQ/flocker/events","assignees_url":"https://api.github.com/repos/ClusterHQ/flocker/assignees{/user}","branches_url":"https://api.github.com/repos/ClusterHQ/flocker/branches{/branch}","tags_url":"https://api.github.com/repos/ClusterHQ/flocker/tags","blobs_url":"https://api.github.com/repos/ClusterHQ/flocker/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ClusterHQ/flocker/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ClusterHQ/flocker/git/refs{/sha}","trees_url":"https://api.github.com/repos/ClusterHQ/flocker/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ClusterHQ/flocker/statuses/{sha}","languages_url":"https://api.github.com/repos/ClusterHQ/flocker/languages","stargazers_url":"https://api.github.com/repos/ClusterHQ/flocker/stargazers","contributors_url":"https://api.github.com/repos/ClusterHQ/flocker/contributors","subscribers_url":"https://api.github.com/repos/ClusterHQ/flocker/subscribers","subscription_url":"https://api.github.com/repos/ClusterHQ/flocker/subscription","commits_url":"https://api.github.com/repos/ClusterHQ/flocker/commits{/sha}","git_commits_url":"https://api.github.com/repos/ClusterHQ/flocker/git/commits{/sha}","comments_url":"https://api.github.com/repos/ClusterHQ/flocker/comments{/number}","issue_comment_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/comments{/number}","contents_url":"https://api.github.com/repos/ClusterHQ/flocker/contents/{+path}","compare_url":"https://api.github.com/repos/ClusterHQ/flocker/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ClusterHQ/flocker/merges","archive_url":"https://api.github.com/repos/ClusterHQ/flocker/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ClusterHQ/flocker/downloads","issues_url":"https://api.github.com/repos/ClusterHQ/flocker/issues{/number}","pulls_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls{/number}","milestones_url":"https://api.github.com/repos/ClusterHQ/flocker/milestones{/number}","notifications_url":"https://api.github.com/repos/ClusterHQ/flocker/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ClusterHQ/flocker/labels{/name}","releases_url":"https://api.github.com/repos/ClusterHQ/flocker/releases{/id}","created_at":"2014-04-28T18:02:14Z","updated_at":"2015-11-11T17:41:02Z","pushed_at":"2015-11-12T14:53:14Z","git_url":"git://github.com/ClusterHQ/flocker.git","ssh_url":"git@github.com:ClusterHQ/flocker.git","clone_url":"https://github.com/ClusterHQ/flocker.git","svn_url":"https://github.com/ClusterHQ/flocker","homepage":"https://clusterhq.com","size":71220,"stargazers_count":1945,"watchers_count":1945,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":142,"mirror_url":null,"open_issues_count":30,"forks":142,"open_issues":30,"watchers":1945,"default_branch":"master"}},"base":{"label":"ClusterHQ:master","ref":"master","sha":"83de9ee0884140d4f66a603e1cc0b8f2962d05c7","user":{"login":"ClusterHQ","id":7913449,"avatar_url":"https://avatars.githubusercontent.com/u/7913449?v=3","gravatar_id":"","url":"https://api.github.com/users/ClusterHQ","html_url":"https://github.com/ClusterHQ","followers_url":"https://api.github.com/users/ClusterHQ/followers","following_url":"https://api.github.com/users/ClusterHQ/following{/other_user}","gists_url":"https://api.github.com/users/ClusterHQ/gists{/gist_id}","starred_url":"https://api.github.com/users/ClusterHQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClusterHQ/subscriptions","organizations_url":"https://api.github.com/users/ClusterHQ/orgs","repos_url":"https://api.github.com/users/ClusterHQ/repos","events_url":"https://api.github.com/users/ClusterHQ/events{/privacy}","received_events_url":"https://api.github.com/users/ClusterHQ/received_events","type":"Organization","site_admin":false},"repo":{"id":19248542,"name":"flocker","full_name":"ClusterHQ/flocker","owner":{"login":"ClusterHQ","id":7913449,"avatar_url":"https://avatars.githubusercontent.com/u/7913449?v=3","gravatar_id":"","url":"https://api.github.com/users/ClusterHQ","html_url":"https://github.com/ClusterHQ","followers_url":"https://api.github.com/users/ClusterHQ/followers","following_url":"https://api.github.com/users/ClusterHQ/following{/other_user}","gists_url":"https://api.github.com/users/ClusterHQ/gists{/gist_id}","starred_url":"https://api.github.com/users/ClusterHQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ClusterHQ/subscriptions","organizations_url":"https://api.github.com/users/ClusterHQ/orgs","repos_url":"https://api.github.com/users/ClusterHQ/repos","events_url":"https://api.github.com/users/ClusterHQ/events{/privacy}","received_events_url":"https://api.github.com/users/ClusterHQ/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ClusterHQ/flocker","description":"Container data volume manager for your Dockerized application","fork":false,"url":"https://api.github.com/repos/ClusterHQ/flocker","forks_url":"https://api.github.com/repos/ClusterHQ/flocker/forks","keys_url":"https://api.github.com/repos/ClusterHQ/flocker/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ClusterHQ/flocker/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ClusterHQ/flocker/teams","hooks_url":"https://api.github.com/repos/ClusterHQ/flocker/hooks","issue_events_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/events{/number}","events_url":"https://api.github.com/repos/ClusterHQ/flocker/events","assignees_url":"https://api.github.com/repos/ClusterHQ/flocker/assignees{/user}","branches_url":"https://api.github.com/repos/ClusterHQ/flocker/branches{/branch}","tags_url":"https://api.github.com/repos/ClusterHQ/flocker/tags","blobs_url":"https://api.github.com/repos/ClusterHQ/flocker/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ClusterHQ/flocker/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ClusterHQ/flocker/git/refs{/sha}","trees_url":"https://api.github.com/repos/ClusterHQ/flocker/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ClusterHQ/flocker/statuses/{sha}","languages_url":"https://api.github.com/repos/ClusterHQ/flocker/languages","stargazers_url":"https://api.github.com/repos/ClusterHQ/flocker/stargazers","contributors_url":"https://api.github.com/repos/ClusterHQ/flocker/contributors","subscribers_url":"https://api.github.com/repos/ClusterHQ/flocker/subscribers","subscription_url":"https://api.github.com/repos/ClusterHQ/flocker/subscription","commits_url":"https://api.github.com/repos/ClusterHQ/flocker/commits{/sha}","git_commits_url":"https://api.github.com/repos/ClusterHQ/flocker/git/commits{/sha}","comments_url":"https://api.github.com/repos/ClusterHQ/flocker/comments{/number}","issue_comment_url":"https://api.github.com/repos/ClusterHQ/flocker/issues/comments{/number}","contents_url":"https://api.github.com/repos/ClusterHQ/flocker/contents/{+path}","compare_url":"https://api.github.com/repos/ClusterHQ/flocker/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ClusterHQ/flocker/merges","archive_url":"https://api.github.com/repos/ClusterHQ/flocker/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ClusterHQ/flocker/downloads","issues_url":"https://api.github.com/repos/ClusterHQ/flocker/issues{/number}","pulls_url":"https://api.github.com/repos/ClusterHQ/flocker/pulls{/number}","milestones_url":"https://api.github.com/repos/ClusterHQ/flocker/milestones{/number}","notifications_url":"https://api.github.com/repos/ClusterHQ/flocker/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ClusterHQ/flocker/labels{/name}","releases_url":"https://api.github.com/repos/ClusterHQ/flocker/releases{/id}","created_at":"2014-04-28T18:02:14Z","updated_at":"2015-11-11T17:41:02Z","pushed_at":"2015-11-12T14:53:14Z","git_url":"git://github.com/ClusterHQ/flocker.git","ssh_url":"git@github.com:ClusterHQ/flocker.git","clone_url":"https://github.com/ClusterHQ/flocker.git","svn_url":"https://github.com/ClusterHQ/flocker","homepage":"https://clusterhq.com","size":71220,"stargazers_count":1945,"watchers_count":1945,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":142,"mirror_url":null,"open_issues_count":30,"forks":142,"open_issues":30,"watchers":1945,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2163"},"html":{"href":"https://github.com/ClusterHQ/flocker/pull/2163"},"issue":{"href":"https://api.github.com/repos/ClusterHQ/flocker/issues/2163"},"comments":{"href":"https://api.github.com/repos/ClusterHQ/flocker/issues/2163/comments"},"review_comments":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2163/comments"},"review_comment":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/ClusterHQ/flocker/pulls/2163/commits"},"statuses":{"href":"https://api.github.com/repos/ClusterHQ/flocker/statuses/325ec0cfd2dc8a70ff9b64b08576c5e88a413f32"}}}}