{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4048","labels_url":"https://api.github.com/repos/apache/spark/issues/4048/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4048/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4048/events","html_url":"https://github.com/apache/spark/pull/4048","id":54373790,"number":4048,"title":"SPARK-4746 make it easy to skip IntegrationTests","user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":18,"created_at":"2015-01-14T20:42:07Z","updated_at":"2015-01-16T06:13:54Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4048","html_url":"https://github.com/apache/spark/pull/4048","diff_url":"https://github.com/apache/spark/pull/4048.diff","patch_url":"https://github.com/apache/spark/pull/4048.patch"},"body":"* create an `IntegrationTest` tag\r\n* label all tests in core as an `IntegrationTest` if they use a `local-cluster`\r\n* make a `unit-test` task in sbt so its easy to skip all the unit tests in local development.\r\n\r\nOn my laptop, this means that I can run `~unit-test` in my sbt console, which takes ~5 mins on the first run.  But since it is calling `test-quick` under the hood, this means that as I make changes, it only re-runs the tests I've effected.  so generally I can update on all tests in a second or two.\r\n\r\nOf course this means its skipping a bunch of important tests, but hopefully this is a useful subset of tests that can actually be run locally.  If you don't skip the IntegrationTests, its totally impractical to ever get through even the first run of `test-quick` on my laptop.\r\n\r\nAn added bonus is that this set of tests can be run without having to ever do the `mvn package` step, since we are never launching a full cluster as an external process."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/70212728","html_url":"https://github.com/apache/spark/pull/4048#issuecomment-70212728","issue_url":"https://api.github.com/repos/apache/spark/issues/4048","id":70212728,"user":{"login":"pwendell","id":320616,"avatar_url":"https://avatars.githubusercontent.com/u/320616?v=3","gravatar_id":"","url":"https://api.github.com/users/pwendell","html_url":"https://github.com/pwendell","followers_url":"https://api.github.com/users/pwendell/followers","following_url":"https://api.github.com/users/pwendell/following{/other_user}","gists_url":"https://api.github.com/users/pwendell/gists{/gist_id}","starred_url":"https://api.github.com/users/pwendell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pwendell/subscriptions","organizations_url":"https://api.github.com/users/pwendell/orgs","repos_url":"https://api.github.com/users/pwendell/repos","events_url":"https://api.github.com/users/pwendell/events{/privacy}","received_events_url":"https://api.github.com/users/pwendell/received_events","type":"User","site_admin":false},"created_at":"2015-01-16T06:13:54Z","updated_at":"2015-01-16T06:13:54Z","body":"@squito another thing is that we should look at whether these tests really need to be integration style tests or not. I've seen people often use `local-cluster` because it's easier to write as an integration test, even if they just put more effort in they could write a proper unit test.\r\n\r\nAlternatively we could try and see if we can run these `local-cluster` tests more quickly in general. I don't think anyone has tried to optimize the execution time of that test path much.\r\n\r\nIn an ideal world the vast majority of our tests would actually be unit tests - i.e. they would run in milliseconds, so that we can have thousands of tests and have them run in a reasonable amount of time."}}