{"action":"created","comment":{"url":"https://api.github.com/repos/CommBank/maestro/pulls/comments/31585007","id":31585007,"diff_hunk":"@@ -50,7 +50,7 @@ With Maestro we are trying to resolve some of these differences, or at least mak\n * Given the awkward support for SQL in the Hadoop world, we need to be able to safely refactor out SQL that needs to be manually optimised using a more powerful API/DSL like [Scalding](https://github.com/twitter/scalding). \n * We don't believe that every data load should choose which storage format to use. A wide array of storage formats is bad from a tooling and delivery coupling perspective. Therefore, Maestro has chosen to support Parquet and Thrift as first class citizens, and all other formats as integration concerns. \n * High level orchestration is actually a bad thing, and so we support orchestration where it makes sense for an atomic unit of work. However, we believe that reactive scheduling makes more sense in terms of flexibility, robustness and decoupling.\n-* Continuous integration and continuous delivery are as important to data application as they are to the frameworks on which these applications are based. Our data developers are required to write automated tests using Maestro provided API. This is vital to allows us to maintain macro level agility of our big data platform. This point is often not well understood, but essentially if you can't automatically validate that you haven't introduced regressions when making change, change quickly stops because it is too slow/risky or costly. We can safely upgrade our Hadoop version (or anything else), run the builds and know what we have broken, and what it should be doing.   \n+* Continuous integration and continuous delivery are as important to data application as they are to the frameworks on which these applications are based. Our data developers are required to write automated tests using Maestro provided API. This is vital as it allows us to maintain macro level agility of our big data platform. This point is often not well understood, but essentially if you can't automatically validate that you haven't introduced regressions when making change, change quickly stops because it is too slow/risky or costly. We can safely upgrade our Hadoop version (or anything else), run the builds and know what we have broken, and what it should be doing.   ","path":"doc/design.md","position":5,"original_position":5,"commit_id":"627a7b4f84ded234d11af7f5be2b41e544b4206a","original_commit_id":"627a7b4f84ded234d11af7f5be2b41e544b4206a","user":{"login":"pdericson","id":2298137,"avatar_url":"https://avatars.githubusercontent.com/u/2298137?v=3","gravatar_id":"","url":"https://api.github.com/users/pdericson","html_url":"https://github.com/pdericson","followers_url":"https://api.github.com/users/pdericson/followers","following_url":"https://api.github.com/users/pdericson/following{/other_user}","gists_url":"https://api.github.com/users/pdericson/gists{/gist_id}","starred_url":"https://api.github.com/users/pdericson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pdericson/subscriptions","organizations_url":"https://api.github.com/users/pdericson/orgs","repos_url":"https://api.github.com/users/pdericson/repos","events_url":"https://api.github.com/users/pdericson/events{/privacy}","received_events_url":"https://api.github.com/users/pdericson/received_events","type":"User","site_admin":false},"body":"*This is vital as it allows us to*","created_at":"2015-06-03T00:29:55Z","updated_at":"2015-06-03T00:29:55Z","html_url":"https://github.com/CommBank/maestro/pull/406#discussion_r31585007","pull_request_url":"https://api.github.com/repos/CommBank/maestro/pulls/406","_links":{"self":{"href":"https://api.github.com/repos/CommBank/maestro/pulls/comments/31585007"},"html":{"href":"https://github.com/CommBank/maestro/pull/406#discussion_r31585007"},"pull_request":{"href":"https://api.github.com/repos/CommBank/maestro/pulls/406"}}},"pull_request":{"url":"https://api.github.com/repos/CommBank/maestro/pulls/406","id":36788447,"html_url":"https://github.com/CommBank/maestro/pull/406","diff_url":"https://github.com/CommBank/maestro/pull/406.diff","patch_url":"https://github.com/CommBank/maestro/pull/406.patch","issue_url":"https://api.github.com/repos/CommBank/maestro/issues/406","number":406,"state":"open","locked":false,"title":"Some minor grammer fixes","user":{"login":"pdericson","id":2298137,"avatar_url":"https://avatars.githubusercontent.com/u/2298137?v=3","gravatar_id":"","url":"https://api.github.com/users/pdericson","html_url":"https://github.com/pdericson","followers_url":"https://api.github.com/users/pdericson/followers","following_url":"https://api.github.com/users/pdericson/following{/other_user}","gists_url":"https://api.github.com/users/pdericson/gists{/gist_id}","starred_url":"https://api.github.com/users/pdericson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pdericson/subscriptions","organizations_url":"https://api.github.com/users/pdericson/orgs","repos_url":"https://api.github.com/users/pdericson/repos","events_url":"https://api.github.com/users/pdericson/events{/privacy}","received_events_url":"https://api.github.com/users/pdericson/received_events","type":"User","site_admin":false},"body":"","created_at":"2015-06-03T00:28:28Z","updated_at":"2015-06-03T00:29:55Z","closed_at":null,"merged_at":null,"merge_commit_sha":"d2cb9afd031ca6a5a09a71915b098364ee6d7ece","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/CommBank/maestro/pulls/406/commits","review_comments_url":"https://api.github.com/repos/CommBank/maestro/pulls/406/comments","review_comment_url":"https://api.github.com/repos/CommBank/maestro/pulls/comments{/number}","comments_url":"https://api.github.com/repos/CommBank/maestro/issues/406/comments","statuses_url":"https://api.github.com/repos/CommBank/maestro/statuses/627a7b4f84ded234d11af7f5be2b41e544b4206a","head":{"label":"pdericson:master","ref":"master","sha":"627a7b4f84ded234d11af7f5be2b41e544b4206a","user":{"login":"pdericson","id":2298137,"avatar_url":"https://avatars.githubusercontent.com/u/2298137?v=3","gravatar_id":"","url":"https://api.github.com/users/pdericson","html_url":"https://github.com/pdericson","followers_url":"https://api.github.com/users/pdericson/followers","following_url":"https://api.github.com/users/pdericson/following{/other_user}","gists_url":"https://api.github.com/users/pdericson/gists{/gist_id}","starred_url":"https://api.github.com/users/pdericson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pdericson/subscriptions","organizations_url":"https://api.github.com/users/pdericson/orgs","repos_url":"https://api.github.com/users/pdericson/repos","events_url":"https://api.github.com/users/pdericson/events{/privacy}","received_events_url":"https://api.github.com/users/pdericson/received_events","type":"User","site_admin":false},"repo":{"id":36729322,"name":"maestro","full_name":"pdericson/maestro","owner":{"login":"pdericson","id":2298137,"avatar_url":"https://avatars.githubusercontent.com/u/2298137?v=3","gravatar_id":"","url":"https://api.github.com/users/pdericson","html_url":"https://github.com/pdericson","followers_url":"https://api.github.com/users/pdericson/followers","following_url":"https://api.github.com/users/pdericson/following{/other_user}","gists_url":"https://api.github.com/users/pdericson/gists{/gist_id}","starred_url":"https://api.github.com/users/pdericson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pdericson/subscriptions","organizations_url":"https://api.github.com/users/pdericson/orgs","repos_url":"https://api.github.com/users/pdericson/repos","events_url":"https://api.github.com/users/pdericson/events{/privacy}","received_events_url":"https://api.github.com/users/pdericson/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/pdericson/maestro","description":"Well-typed data pipeline framework for building robust ETL jobs","fork":true,"url":"https://api.github.com/repos/pdericson/maestro","forks_url":"https://api.github.com/repos/pdericson/maestro/forks","keys_url":"https://api.github.com/repos/pdericson/maestro/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pdericson/maestro/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pdericson/maestro/teams","hooks_url":"https://api.github.com/repos/pdericson/maestro/hooks","issue_events_url":"https://api.github.com/repos/pdericson/maestro/issues/events{/number}","events_url":"https://api.github.com/repos/pdericson/maestro/events","assignees_url":"https://api.github.com/repos/pdericson/maestro/assignees{/user}","branches_url":"https://api.github.com/repos/pdericson/maestro/branches{/branch}","tags_url":"https://api.github.com/repos/pdericson/maestro/tags","blobs_url":"https://api.github.com/repos/pdericson/maestro/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pdericson/maestro/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pdericson/maestro/git/refs{/sha}","trees_url":"https://api.github.com/repos/pdericson/maestro/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pdericson/maestro/statuses/{sha}","languages_url":"https://api.github.com/repos/pdericson/maestro/languages","stargazers_url":"https://api.github.com/repos/pdericson/maestro/stargazers","contributors_url":"https://api.github.com/repos/pdericson/maestro/contributors","subscribers_url":"https://api.github.com/repos/pdericson/maestro/subscribers","subscription_url":"https://api.github.com/repos/pdericson/maestro/subscription","commits_url":"https://api.github.com/repos/pdericson/maestro/commits{/sha}","git_commits_url":"https://api.github.com/repos/pdericson/maestro/git/commits{/sha}","comments_url":"https://api.github.com/repos/pdericson/maestro/comments{/number}","issue_comment_url":"https://api.github.com/repos/pdericson/maestro/issues/comments{/number}","contents_url":"https://api.github.com/repos/pdericson/maestro/contents/{+path}","compare_url":"https://api.github.com/repos/pdericson/maestro/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pdericson/maestro/merges","archive_url":"https://api.github.com/repos/pdericson/maestro/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pdericson/maestro/downloads","issues_url":"https://api.github.com/repos/pdericson/maestro/issues{/number}","pulls_url":"https://api.github.com/repos/pdericson/maestro/pulls{/number}","milestones_url":"https://api.github.com/repos/pdericson/maestro/milestones{/number}","notifications_url":"https://api.github.com/repos/pdericson/maestro/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pdericson/maestro/labels{/name}","releases_url":"https://api.github.com/repos/pdericson/maestro/releases{/id}","created_at":"2015-06-02T11:44:32Z","updated_at":"2015-06-02T11:44:33Z","pushed_at":"2015-06-03T00:25:24Z","git_url":"git://github.com/pdericson/maestro.git","ssh_url":"git@github.com:pdericson/maestro.git","clone_url":"https://github.com/pdericson/maestro.git","svn_url":"https://github.com/pdericson/maestro","homepage":"https://commbank.github.io/maestro","size":48207,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"CommBank:master","ref":"master","sha":"7550b557686432c769d8fafc3b09cf75b376f1ef","user":{"login":"CommBank","id":4240076,"avatar_url":"https://avatars.githubusercontent.com/u/4240076?v=3","gravatar_id":"","url":"https://api.github.com/users/CommBank","html_url":"https://github.com/CommBank","followers_url":"https://api.github.com/users/CommBank/followers","following_url":"https://api.github.com/users/CommBank/following{/other_user}","gists_url":"https://api.github.com/users/CommBank/gists{/gist_id}","starred_url":"https://api.github.com/users/CommBank/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CommBank/subscriptions","organizations_url":"https://api.github.com/users/CommBank/orgs","repos_url":"https://api.github.com/users/CommBank/repos","events_url":"https://api.github.com/users/CommBank/events{/privacy}","received_events_url":"https://api.github.com/users/CommBank/received_events","type":"Organization","site_admin":false},"repo":{"id":17347463,"name":"maestro","full_name":"CommBank/maestro","owner":{"login":"CommBank","id":4240076,"avatar_url":"https://avatars.githubusercontent.com/u/4240076?v=3","gravatar_id":"","url":"https://api.github.com/users/CommBank","html_url":"https://github.com/CommBank","followers_url":"https://api.github.com/users/CommBank/followers","following_url":"https://api.github.com/users/CommBank/following{/other_user}","gists_url":"https://api.github.com/users/CommBank/gists{/gist_id}","starred_url":"https://api.github.com/users/CommBank/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CommBank/subscriptions","organizations_url":"https://api.github.com/users/CommBank/orgs","repos_url":"https://api.github.com/users/CommBank/repos","events_url":"https://api.github.com/users/CommBank/events{/privacy}","received_events_url":"https://api.github.com/users/CommBank/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/CommBank/maestro","description":"Well-typed data pipeline framework for building robust ETL jobs","fork":false,"url":"https://api.github.com/repos/CommBank/maestro","forks_url":"https://api.github.com/repos/CommBank/maestro/forks","keys_url":"https://api.github.com/repos/CommBank/maestro/keys{/key_id}","collaborators_url":"https://api.github.com/repos/CommBank/maestro/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/CommBank/maestro/teams","hooks_url":"https://api.github.com/repos/CommBank/maestro/hooks","issue_events_url":"https://api.github.com/repos/CommBank/maestro/issues/events{/number}","events_url":"https://api.github.com/repos/CommBank/maestro/events","assignees_url":"https://api.github.com/repos/CommBank/maestro/assignees{/user}","branches_url":"https://api.github.com/repos/CommBank/maestro/branches{/branch}","tags_url":"https://api.github.com/repos/CommBank/maestro/tags","blobs_url":"https://api.github.com/repos/CommBank/maestro/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/CommBank/maestro/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/CommBank/maestro/git/refs{/sha}","trees_url":"https://api.github.com/repos/CommBank/maestro/git/trees{/sha}","statuses_url":"https://api.github.com/repos/CommBank/maestro/statuses/{sha}","languages_url":"https://api.github.com/repos/CommBank/maestro/languages","stargazers_url":"https://api.github.com/repos/CommBank/maestro/stargazers","contributors_url":"https://api.github.com/repos/CommBank/maestro/contributors","subscribers_url":"https://api.github.com/repos/CommBank/maestro/subscribers","subscription_url":"https://api.github.com/repos/CommBank/maestro/subscription","commits_url":"https://api.github.com/repos/CommBank/maestro/commits{/sha}","git_commits_url":"https://api.github.com/repos/CommBank/maestro/git/commits{/sha}","comments_url":"https://api.github.com/repos/CommBank/maestro/comments{/number}","issue_comment_url":"https://api.github.com/repos/CommBank/maestro/issues/comments{/number}","contents_url":"https://api.github.com/repos/CommBank/maestro/contents/{+path}","compare_url":"https://api.github.com/repos/CommBank/maestro/compare/{base}...{head}","merges_url":"https://api.github.com/repos/CommBank/maestro/merges","archive_url":"https://api.github.com/repos/CommBank/maestro/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/CommBank/maestro/downloads","issues_url":"https://api.github.com/repos/CommBank/maestro/issues{/number}","pulls_url":"https://api.github.com/repos/CommBank/maestro/pulls{/number}","milestones_url":"https://api.github.com/repos/CommBank/maestro/milestones{/number}","notifications_url":"https://api.github.com/repos/CommBank/maestro/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/CommBank/maestro/labels{/name}","releases_url":"https://api.github.com/repos/CommBank/maestro/releases{/id}","created_at":"2014-03-02T21:34:53Z","updated_at":"2015-06-01T06:58:13Z","pushed_at":"2015-06-03T00:28:29Z","git_url":"git://github.com/CommBank/maestro.git","ssh_url":"git@github.com:CommBank/maestro.git","clone_url":"https://github.com/CommBank/maestro.git","svn_url":"https://github.com/CommBank/maestro","homepage":"https://commbank.github.io/maestro","size":48207,"stargazers_count":32,"watchers_count":32,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":20,"mirror_url":null,"open_issues_count":57,"forks":20,"open_issues":57,"watchers":32,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/CommBank/maestro/pulls/406"},"html":{"href":"https://github.com/CommBank/maestro/pull/406"},"issue":{"href":"https://api.github.com/repos/CommBank/maestro/issues/406"},"comments":{"href":"https://api.github.com/repos/CommBank/maestro/issues/406/comments"},"review_comments":{"href":"https://api.github.com/repos/CommBank/maestro/pulls/406/comments"},"review_comment":{"href":"https://api.github.com/repos/CommBank/maestro/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/CommBank/maestro/pulls/406/commits"},"statuses":{"href":"https://api.github.com/repos/CommBank/maestro/statuses/627a7b4f84ded234d11af7f5be2b41e544b4206a"}}}}