{"action":"created","issue":{"url":"https://api.github.com/repos/twitter/scalding/issues/1220","labels_url":"https://api.github.com/repos/twitter/scalding/issues/1220/labels{/name}","comments_url":"https://api.github.com/repos/twitter/scalding/issues/1220/comments","events_url":"https://api.github.com/repos/twitter/scalding/issues/1220/events","html_url":"https://github.com/twitter/scalding/pull/1220","id":60078426,"number":1220,"title":"Make Cascading Fabric selection configurable, no longer hardcoding Hadoo...","user":{"login":"cchepelov","id":1515440,"avatar_url":"https://avatars.githubusercontent.com/u/1515440?v=3","gravatar_id":"","url":"https://api.github.com/users/cchepelov","html_url":"https://github.com/cchepelov","followers_url":"https://api.github.com/users/cchepelov/followers","following_url":"https://api.github.com/users/cchepelov/following{/other_user}","gists_url":"https://api.github.com/users/cchepelov/gists{/gist_id}","starred_url":"https://api.github.com/users/cchepelov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cchepelov/subscriptions","organizations_url":"https://api.github.com/users/cchepelov/orgs","repos_url":"https://api.github.com/users/cchepelov/repos","events_url":"https://api.github.com/users/cchepelov/events{/privacy}","received_events_url":"https://api.github.com/users/cchepelov/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-03-06T09:29:17Z","updated_at":"2015-05-06T16:55:12Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/twitter/scalding/pulls/1220","html_url":"https://github.com/twitter/scalding/pull/1220","diff_url":"https://github.com/twitter/scalding/pull/1220.diff","patch_url":"https://github.com/twitter/scalding/pull/1220.patch"},"body":"This removes hardcoding for HadoopFlowConnector in c.t.s.Mode, and adds in new job configuration parameters to allow for supplying the desired class instead.\r\n\r\nAlso, the --hdfs parameter is complemented with --hadoop2 (same behaviour), --hadoop2-mr1 and --hadoop2-tez which supply implementation names for FlowConnector and FlowProcess\r\n\r\n--local remains hard-wired to LocalFlowConnector\r\n"},"comment":{"url":"https://api.github.com/repos/twitter/scalding/issues/comments/99536719","html_url":"https://github.com/twitter/scalding/pull/1220#issuecomment-99536719","issue_url":"https://api.github.com/repos/twitter/scalding/issues/1220","id":99536719,"user":{"login":"cchepelov","id":1515440,"avatar_url":"https://avatars.githubusercontent.com/u/1515440?v=3","gravatar_id":"","url":"https://api.github.com/users/cchepelov","html_url":"https://github.com/cchepelov","followers_url":"https://api.github.com/users/cchepelov/followers","following_url":"https://api.github.com/users/cchepelov/following{/other_user}","gists_url":"https://api.github.com/users/cchepelov/gists{/gist_id}","starred_url":"https://api.github.com/users/cchepelov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cchepelov/subscriptions","organizations_url":"https://api.github.com/users/cchepelov/orgs","repos_url":"https://api.github.com/users/cchepelov/repos","events_url":"https://api.github.com/users/cchepelov/events{/privacy}","received_events_url":"https://api.github.com/users/cchepelov/received_events","type":"User","site_admin":false},"created_at":"2015-05-06T16:55:12Z","updated_at":"2015-05-06T16:55:12Z","body":"Indeed no, I haven't tried to run the scalding tests against cascading\n3; wanted to patch with as light a touch as I could. However, the patch\nis also valid on a pure Cascading 2.6.X perspective: it enables running\nwith the /hadoop2-mr1/ fabric rather than the legacy /hadoop/ fabric.\nAnd the tests did pass, at least when I last updated the PR.\n\nAs I understand it, enabling fabrics and upgrading the version of the\nCascading frameworks involved are quite orthogonal undertakings. This PR\nonly enables the selection of the fabric, bumping cascading's major\nversion remains an application-level responsibility (thankfully, the API\nsubset used by Scalding hasn't changed).\n\nWith respect to Cascading 3.0.0-wip-X and Scalding (whichever the\nbackend between local, hadoop, hadoop2-mr1 and hadoop2-tez): we're now\nrunning all our application-level acceptance tests on all four backends\nat all times (a milestone which we hadn't achieved a month ago when I\nlast updated this PR). This has since my initial public announcement\nuncovered a few subtle nasty behaviours, which we're reporting on a\nreal-time basis to @cwensel. We have almost-daily exchanges of reports\nand updated -wips, and from our consumer perspective, the progress is great.\nI wonder if it wouldn't make sense to start something automated in\nscalding to track those non-passing scalding tests as the 3.0.0-wip-X\ncreeps towards general release, at least on the -hadoop backend. Again,\nthis is a distinct issue from fabric-switching. Would a separate PR be\nuseful to track cascading-3 compatibility?\n\n\nLe 06/05/2015 17:23, ianoc a écrit :\n>\n> So you've never tried to run the scalding tests against cascading 3\n> then? your going between two major versions of cascading that aren't\n> necessarily binary compatible. (indeed when i tried it tests don't\n> pass, which would be worrysome to merge this).\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/twitter/scalding/pull/1220#issuecomment-99513039>.\n>\n\n"}}