{"action":"created","comment":{"url":"https://api.github.com/repos/Shopify/spark/pulls/comments/24443799","id":24443799,"diff_hunk":"@@ -9,6 +9,9 @@ set :shared_conf_path, \"/u/apps/spark/shared/conf\"\n set :spark_jar_path, \"hdfs://hadoop-production/user/sparkles\"\n set :gateway, nil\n set :keep_releases, 5\n+set :branch, fetch(:branch, `git symbolic-ref --short HEAD`.gsub(\"\\s\",\"\"))","path":"Capfile","position":4,"original_position":4,"commit_id":"15652c8c78d2838ecc940355fd1816fdf37cfed6","original_commit_id":"15652c8c78d2838ecc940355fd1816fdf37cfed6","user":{"login":"orenmazor","id":384279,"avatar_url":"https://avatars.githubusercontent.com/u/384279?v=3","gravatar_id":"","url":"https://api.github.com/users/orenmazor","html_url":"https://github.com/orenmazor","followers_url":"https://api.github.com/users/orenmazor/followers","following_url":"https://api.github.com/users/orenmazor/following{/other_user}","gists_url":"https://api.github.com/users/orenmazor/gists{/gist_id}","starred_url":"https://api.github.com/users/orenmazor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/orenmazor/subscriptions","organizations_url":"https://api.github.com/users/orenmazor/orgs","repos_url":"https://api.github.com/users/orenmazor/repos","events_url":"https://api.github.com/users/orenmazor/events{/privacy}","received_events_url":"https://api.github.com/users/orenmazor/received_events","type":"User","site_admin":false},"body":"this does not get set anywhere because we dont use cap environments. need this to detect whether we're on master or on a feature branch doing tests.","created_at":"2015-02-10T19:44:19Z","updated_at":"2015-02-10T19:44:19Z","html_url":"https://github.com/Shopify/spark/pull/51#discussion_r24443799","pull_request_url":"https://api.github.com/repos/Shopify/spark/pulls/51","_links":{"self":{"href":"https://api.github.com/repos/Shopify/spark/pulls/comments/24443799"},"html":{"href":"https://github.com/Shopify/spark/pull/51#discussion_r24443799"},"pull_request":{"href":"https://api.github.com/repos/Shopify/spark/pulls/51"}}},"pull_request":{"url":"https://api.github.com/repos/Shopify/spark/pulls/51","id":29037918,"html_url":"https://github.com/Shopify/spark/pull/51","diff_url":"https://github.com/Shopify/spark/pull/51.diff","patch_url":"https://github.com/Shopify/spark/pull/51.patch","issue_url":"https://api.github.com/repos/Shopify/spark/issues/51","number":51,"state":"open","locked":false,"title":"Fix test spark jar","user":{"login":"orenmazor","id":384279,"avatar_url":"https://avatars.githubusercontent.com/u/384279?v=3","gravatar_id":"","url":"https://api.github.com/users/orenmazor","html_url":"https://github.com/orenmazor","followers_url":"https://api.github.com/users/orenmazor/followers","following_url":"https://api.github.com/users/orenmazor/following{/other_user}","gists_url":"https://api.github.com/users/orenmazor/gists{/gist_id}","starred_url":"https://api.github.com/users/orenmazor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/orenmazor/subscriptions","organizations_url":"https://api.github.com/users/orenmazor/orgs","repos_url":"https://api.github.com/users/orenmazor/repos","events_url":"https://api.github.com/users/orenmazor/events{/privacy}","received_events_url":"https://api.github.com/users/orenmazor/received_events","type":"User","site_admin":false},"body":"Yesterday I attempted to ship an upstream merge to spark. This merge failed and revealed a primary flaw in our testing, which also revealed a flaw in how we can test.\r\n\r\nThe problem:\r\n\r\noriginally deploy:upload_to_hdfs was uploading the spark jar from /u/apps/spark/current *before* the symlink was updated, which means we were always running deploy:test_spark_jar from an incorrect spark jar (and also running spark from the wrong revision *facepalm*\r\n\r\nThe fix:\r\n\r\nThis went out already and is on master, but the fix is to make sure that when we do an actual deploy, we upload the spark jar from the releases_path, not current_path.\r\n\r\n------\r\n\r\nthat was context. this PR fixes issues involving our ability to actually upload and test arbitrary spark jars from hdfs. \r\n\r\n```deploy:upload_spark_jar``` - updated to detect whether we have a spark assembly that matches the current local sha (not packserv sha). if it is not present, we need to build it.\r\n\r\n```deploy:test_spark_jar``` - test spark jar with the local_sha, not packserv sha.\r\n\r\ndetailed comments inline.","created_at":"2015-02-10T19:43:44Z","updated_at":"2015-02-10T19:44:19Z","closed_at":null,"merged_at":null,"merge_commit_sha":"8976a1f0db706f6b50c75905d151b80631f4602f","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/Shopify/spark/pulls/51/commits","review_comments_url":"https://api.github.com/repos/Shopify/spark/pulls/51/comments","review_comment_url":"https://api.github.com/repos/Shopify/spark/pulls/comments/{number}","comments_url":"https://api.github.com/repos/Shopify/spark/issues/51/comments","statuses_url":"https://api.github.com/repos/Shopify/spark/statuses/15652c8c78d2838ecc940355fd1816fdf37cfed6","head":{"label":"Shopify:fix_test_spark_jar","ref":"fix_test_spark_jar","sha":"15652c8c78d2838ecc940355fd1816fdf37cfed6","user":{"login":"Shopify","id":8085,"avatar_url":"https://avatars.githubusercontent.com/u/8085?v=3","gravatar_id":"","url":"https://api.github.com/users/Shopify","html_url":"https://github.com/Shopify","followers_url":"https://api.github.com/users/Shopify/followers","following_url":"https://api.github.com/users/Shopify/following{/other_user}","gists_url":"https://api.github.com/users/Shopify/gists{/gist_id}","starred_url":"https://api.github.com/users/Shopify/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Shopify/subscriptions","organizations_url":"https://api.github.com/users/Shopify/orgs","repos_url":"https://api.github.com/users/Shopify/repos","events_url":"https://api.github.com/users/Shopify/events{/privacy}","received_events_url":"https://api.github.com/users/Shopify/received_events","type":"Organization","site_admin":false},"repo":{"id":18164102,"name":"spark","full_name":"Shopify/spark","owner":{"login":"Shopify","id":8085,"avatar_url":"https://avatars.githubusercontent.com/u/8085?v=3","gravatar_id":"","url":"https://api.github.com/users/Shopify","html_url":"https://github.com/Shopify","followers_url":"https://api.github.com/users/Shopify/followers","following_url":"https://api.github.com/users/Shopify/following{/other_user}","gists_url":"https://api.github.com/users/Shopify/gists{/gist_id}","starred_url":"https://api.github.com/users/Shopify/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Shopify/subscriptions","organizations_url":"https://api.github.com/users/Shopify/orgs","repos_url":"https://api.github.com/users/Shopify/repos","events_url":"https://api.github.com/users/Shopify/events{/privacy}","received_events_url":"https://api.github.com/users/Shopify/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Shopify/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/Shopify/spark","forks_url":"https://api.github.com/repos/Shopify/spark/forks","keys_url":"https://api.github.com/repos/Shopify/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Shopify/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Shopify/spark/teams","hooks_url":"https://api.github.com/repos/Shopify/spark/hooks","issue_events_url":"https://api.github.com/repos/Shopify/spark/issues/events{/number}","events_url":"https://api.github.com/repos/Shopify/spark/events","assignees_url":"https://api.github.com/repos/Shopify/spark/assignees{/user}","branches_url":"https://api.github.com/repos/Shopify/spark/branches{/branch}","tags_url":"https://api.github.com/repos/Shopify/spark/tags","blobs_url":"https://api.github.com/repos/Shopify/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Shopify/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Shopify/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/Shopify/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Shopify/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/Shopify/spark/languages","stargazers_url":"https://api.github.com/repos/Shopify/spark/stargazers","contributors_url":"https://api.github.com/repos/Shopify/spark/contributors","subscribers_url":"https://api.github.com/repos/Shopify/spark/subscribers","subscription_url":"https://api.github.com/repos/Shopify/spark/subscription","commits_url":"https://api.github.com/repos/Shopify/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/Shopify/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/Shopify/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/Shopify/spark/issues/comments/{number}","contents_url":"https://api.github.com/repos/Shopify/spark/contents/{+path}","compare_url":"https://api.github.com/repos/Shopify/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Shopify/spark/merges","archive_url":"https://api.github.com/repos/Shopify/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Shopify/spark/downloads","issues_url":"https://api.github.com/repos/Shopify/spark/issues{/number}","pulls_url":"https://api.github.com/repos/Shopify/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/Shopify/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/Shopify/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Shopify/spark/labels{/name}","releases_url":"https://api.github.com/repos/Shopify/spark/releases{/id}","created_at":"2014-03-27T04:12:25Z","updated_at":"2015-02-09T19:55:32Z","pushed_at":"2015-02-10T19:38:32Z","git_url":"git://github.com/Shopify/spark.git","ssh_url":"git@github.com:Shopify/spark.git","clone_url":"https://github.com/Shopify/spark.git","svn_url":"https://github.com/Shopify/spark","homepage":null,"size":87058,"stargazers_count":1,"watchers_count":1,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":6,"forks":0,"open_issues":6,"watchers":1,"default_branch":"master"}},"base":{"label":"Shopify:master","ref":"master","sha":"b08930512c90c21ed8953772136900027fd571dd","user":{"login":"Shopify","id":8085,"avatar_url":"https://avatars.githubusercontent.com/u/8085?v=3","gravatar_id":"","url":"https://api.github.com/users/Shopify","html_url":"https://github.com/Shopify","followers_url":"https://api.github.com/users/Shopify/followers","following_url":"https://api.github.com/users/Shopify/following{/other_user}","gists_url":"https://api.github.com/users/Shopify/gists{/gist_id}","starred_url":"https://api.github.com/users/Shopify/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Shopify/subscriptions","organizations_url":"https://api.github.com/users/Shopify/orgs","repos_url":"https://api.github.com/users/Shopify/repos","events_url":"https://api.github.com/users/Shopify/events{/privacy}","received_events_url":"https://api.github.com/users/Shopify/received_events","type":"Organization","site_admin":false},"repo":{"id":18164102,"name":"spark","full_name":"Shopify/spark","owner":{"login":"Shopify","id":8085,"avatar_url":"https://avatars.githubusercontent.com/u/8085?v=3","gravatar_id":"","url":"https://api.github.com/users/Shopify","html_url":"https://github.com/Shopify","followers_url":"https://api.github.com/users/Shopify/followers","following_url":"https://api.github.com/users/Shopify/following{/other_user}","gists_url":"https://api.github.com/users/Shopify/gists{/gist_id}","starred_url":"https://api.github.com/users/Shopify/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Shopify/subscriptions","organizations_url":"https://api.github.com/users/Shopify/orgs","repos_url":"https://api.github.com/users/Shopify/repos","events_url":"https://api.github.com/users/Shopify/events{/privacy}","received_events_url":"https://api.github.com/users/Shopify/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Shopify/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/Shopify/spark","forks_url":"https://api.github.com/repos/Shopify/spark/forks","keys_url":"https://api.github.com/repos/Shopify/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Shopify/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Shopify/spark/teams","hooks_url":"https://api.github.com/repos/Shopify/spark/hooks","issue_events_url":"https://api.github.com/repos/Shopify/spark/issues/events{/number}","events_url":"https://api.github.com/repos/Shopify/spark/events","assignees_url":"https://api.github.com/repos/Shopify/spark/assignees{/user}","branches_url":"https://api.github.com/repos/Shopify/spark/branches{/branch}","tags_url":"https://api.github.com/repos/Shopify/spark/tags","blobs_url":"https://api.github.com/repos/Shopify/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Shopify/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Shopify/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/Shopify/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Shopify/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/Shopify/spark/languages","stargazers_url":"https://api.github.com/repos/Shopify/spark/stargazers","contributors_url":"https://api.github.com/repos/Shopify/spark/contributors","subscribers_url":"https://api.github.com/repos/Shopify/spark/subscribers","subscription_url":"https://api.github.com/repos/Shopify/spark/subscription","commits_url":"https://api.github.com/repos/Shopify/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/Shopify/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/Shopify/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/Shopify/spark/issues/comments/{number}","contents_url":"https://api.github.com/repos/Shopify/spark/contents/{+path}","compare_url":"https://api.github.com/repos/Shopify/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Shopify/spark/merges","archive_url":"https://api.github.com/repos/Shopify/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Shopify/spark/downloads","issues_url":"https://api.github.com/repos/Shopify/spark/issues{/number}","pulls_url":"https://api.github.com/repos/Shopify/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/Shopify/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/Shopify/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Shopify/spark/labels{/name}","releases_url":"https://api.github.com/repos/Shopify/spark/releases{/id}","created_at":"2014-03-27T04:12:25Z","updated_at":"2015-02-09T19:55:32Z","pushed_at":"2015-02-10T19:38:32Z","git_url":"git://github.com/Shopify/spark.git","ssh_url":"git@github.com:Shopify/spark.git","clone_url":"https://github.com/Shopify/spark.git","svn_url":"https://github.com/Shopify/spark","homepage":null,"size":87058,"stargazers_count":1,"watchers_count":1,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":6,"forks":0,"open_issues":6,"watchers":1,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/Shopify/spark/pulls/51"},"html":{"href":"https://github.com/Shopify/spark/pull/51"},"issue":{"href":"https://api.github.com/repos/Shopify/spark/issues/51"},"comments":{"href":"https://api.github.com/repos/Shopify/spark/issues/51/comments"},"review_comments":{"href":"https://api.github.com/repos/Shopify/spark/pulls/51/comments"},"review_comment":{"href":"https://api.github.com/repos/Shopify/spark/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/Shopify/spark/pulls/51/commits"},"statuses":{"href":"https://api.github.com/repos/Shopify/spark/statuses/15652c8c78d2838ecc940355fd1816fdf37cfed6"}}}}