{"action":"created","issue":{"url":"https://api.github.com/repos/druid-io/druid/issues/1177","labels_url":"https://api.github.com/repos/druid-io/druid/issues/1177/labels{/name}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/1177/comments","events_url":"https://api.github.com/repos/druid-io/druid/issues/1177/events","html_url":"https://github.com/druid-io/druid/pull/1177","id":60214621,"number":1177,"title":"Proposal: feature to make hadoop input format configurable for batch ingestion","user":{"login":"himanshug","id":246308,"avatar_url":"https://avatars.githubusercontent.com/u/246308?v=3","gravatar_id":"","url":"https://api.github.com/users/himanshug","html_url":"https://github.com/himanshug","followers_url":"https://api.github.com/users/himanshug/followers","following_url":"https://api.github.com/users/himanshug/following{/other_user}","gists_url":"https://api.github.com/users/himanshug/gists{/gist_id}","starred_url":"https://api.github.com/users/himanshug/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/himanshug/subscriptions","organizations_url":"https://api.github.com/users/himanshug/orgs","repos_url":"https://api.github.com/users/himanshug/repos","events_url":"https://api.github.com/users/himanshug/events{/privacy}","received_events_url":"https://api.github.com/users/himanshug/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-03-07T17:02:55Z","updated_at":"2015-03-18T21:11:01Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/druid-io/druid/pulls/1177","html_url":"https://github.com/druid-io/druid/pull/1177","diff_url":"https://github.com/druid-io/druid/pull/1177.diff","patch_url":"https://github.com/druid-io/druid/pull/1177.patch"},"body":"Our hadooop batch pipelines mostly store data using avro input format. Currently we need to run additional jobs just to convert avro data into text(csv/json/tsv) so that it can be ingested into druid. This is really not necessary. We can save those hadoop grid resources and additional jobs by letting users specify their own hadoop input format for the batch ingestion to druid.\r\n\r\nThis patch enables additional parameter inside hadoop ioConfig as following..\r\n```\r\n \"ioConfig\" : {\r\n    \"type\" : \"hadoop\",\r\n    \"inputSpec\" : {\r\n      \"type\" : \"static\",\r\n      \"paths\" : \"/MyDirectory/examples/indexing/avrodata\",\r\n      \"inputFormat\": \"com.xyz.hadoop.MyAvroToTextInputFormat\"\r\n    }\r\n```\r\n\r\n\r\nIt is required that user supplied InputFormat reads input data location from \"mapreduce.input.fileinputformat.inputdir\" in the job configuration.\r\n\r\nand it must extend from \"org.apache.hadoop.mapreduce.InputFormat\"\r\n"},"comment":{"url":"https://api.github.com/repos/druid-io/druid/issues/comments/83178188","html_url":"https://github.com/druid-io/druid/pull/1177#issuecomment-83178188","issue_url":"https://api.github.com/repos/druid-io/druid/issues/1177","id":83178188,"user":{"login":"himanshug","id":246308,"avatar_url":"https://avatars.githubusercontent.com/u/246308?v=3","gravatar_id":"","url":"https://api.github.com/users/himanshug","html_url":"https://github.com/himanshug","followers_url":"https://api.github.com/users/himanshug/followers","following_url":"https://api.github.com/users/himanshug/following{/other_user}","gists_url":"https://api.github.com/users/himanshug/gists{/gist_id}","starred_url":"https://api.github.com/users/himanshug/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/himanshug/subscriptions","organizations_url":"https://api.github.com/users/himanshug/orgs","repos_url":"https://api.github.com/users/himanshug/repos","events_url":"https://api.github.com/users/himanshug/events{/privacy}","received_events_url":"https://api.github.com/users/himanshug/received_events","type":"User","site_admin":false},"created_at":"2015-03-18T21:11:01Z","updated_at":"2015-03-18T21:11:01Z","body":"@xvrl this PR is complete and ready for merging. can u pls see?"}}