{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/405","labels_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/405/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/405/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/405/events","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/405","id":64586963,"number":405,"title":"Es-Hadoop ingestion through Pig is missing the mappings","user":{"login":"BalachanderGS","id":10089328,"avatar_url":"https://avatars.githubusercontent.com/u/10089328?v=3","gravatar_id":"","url":"https://api.github.com/users/BalachanderGS","html_url":"https://github.com/BalachanderGS","followers_url":"https://api.github.com/users/BalachanderGS/followers","following_url":"https://api.github.com/users/BalachanderGS/following{/other_user}","gists_url":"https://api.github.com/users/BalachanderGS/gists{/gist_id}","starred_url":"https://api.github.com/users/BalachanderGS/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BalachanderGS/subscriptions","organizations_url":"https://api.github.com/users/BalachanderGS/orgs","repos_url":"https://api.github.com/users/BalachanderGS/repos","events_url":"https://api.github.com/users/BalachanderGS/events{/privacy}","received_events_url":"https://api.github.com/users/BalachanderGS/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-03-26T17:22:43Z","updated_at":"2015-03-30T16:00:45Z","closed_at":null,"body":"ES:\r\n\r\ncurl -XDELETE 'http://localhost:9200/ztmp_inventory_tool_sample'\r\n\r\ncurl -XPOST localhost:9200/ztmp_inventory_tool_sample -d '{\r\n\"settings\" : {\r\n\"term_index_interval\" : 256,\r\n\"term_index_divisor\" : 5\r\n},\r\n\"mappings\" : {\r\n\"invData\" : {\r\n\"_source\" : { \"enabled\" : true },\r\n\"properties\" : {\r\n\"ekv_raw\" : { \"type\" : \"byte\" },\r\n\"ekv_flight\" : { \"type\" : \"byte\" },\r\n\"event_id\" : { \"type\" : \"long\" },\r\n\"cookie_id\" : { \"type\" : \"long\" },\r\n\"dpId\" : { \"type\" : \"short\" },\r\n\"vertical\" : { \"type\" : \"string\" },\r\n\"activity_group\" : { \"type\" : \"string\" },\r\n\"activity\" : { \"type\" : \"string\" },\r\n\"eventDateTime\" : {\"type\":\"date\", \"format\":\"YYYY-MM-dd'\"'T'\"'HH:mm:ss.SSSZ\"},\r\n\"departureDate\" : { \"type\" : \"date\", \"format\":\"YYYY-MM-dd\", \"ignore_malformed\" : true},\r\n\"returnDate\" : { \"type\" : \"string\" },\r\n\"origin\" : { \"type\" : \"string\" },\r\n\"destination\" : { \"type\" : \"string\" },\r\n\"destination_country_code\" : { \"type\" : \"string\" },\r\n\"destination_state\" : { \"type\" : \"string\" },\r\n\"destination_city\" : { \"type\" : \"string\" },\r\n\"carrier\" : { \"type\" : \"string\" },\r\n\"cabinClassGroup\" : { \"type\" : \"string\" },\r\n\"currency\" : { \"type\" : \"string\" },\r\n\"travelers\" : { \"type\" : \"short\" },\r\n\"duration\" : { \"type\" : \"short\" },\r\n\"bookedDate\" : { \"type\" : \"date\", \"format\":\"YYYY-MM-dd\", \"ignore_malformed\" : true},\r\n\"airFare\" : { \"type\" : \"float\" }}\r\n}}}'\r\n\r\nREGISTER elasticsearch-hadoop-2.1.0.Beta3/dist/elasticsearch-hadoop-2.1.0.Beta3.jar\r\n\r\nDEFINE EsStorage org.elasticsearch.hadoop.pig.EsStorage('es.http.timeout=1m',\r\n'es.resource=ztmp_inventory_tool_sample/invData',\r\n'es.mapping.pig.tuple.use.field.names = true',\r\n'es.http.timeout = 5s',\r\n'es.index.auto.create = false',\r\n'es.nodes = 68.67.141.63',\r\n'es.port = 9200');\r\n\r\ntestEs = LOAD '/user/bganapathy/flightPKeys' USING PigStorage('|') AS (ekv_raw: int, ekv_flight: int, eventId: long, cookie_id: long, dpId: int, vertical: chararray, activity_group: chararray, activity: chararray, eventDateTime: chararray, departureDate: chararray, returnDate: chararray, origin: chararray, destination: chararray, destination_country_code: chararray, destination_state: chararray, destination_city: chararray, carrier: chararray, cabinClassGroup: chararray, currency: chararray, travelers: chararray, duration: chararray, bookedDate: chararray, airFare: chararray);\r\n\r\ntestEs = LIMIT testEs 100000;\r\nSTORE testEs INTO 'ztmp_inventory_tool_sample/invData' USING EsStorage();\r\n\r\nLogs:\r\n\r\n2015-03-24 22:10:34,544 [JobControl] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.\r\n2015-03-24 22:10:34,755 [JobControl] INFO org.elasticsearch.hadoop.mr.EsOutputFormat - Writing to [ztmp_inventory_tool_sample/invData]\r\n2015-03-24 22:10:34,774 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1\r\n2015-03-24 22:10:34,774 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1\r\n2015-03-24 22:10:34,776 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1\r\n2015-03-24 22:10:35,971 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201408291723_27582\r\n2015-03-24 22:10:35,971 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases testEs\r\n2015-03-24 22:10:35,971 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: C: R: testEs[-1,-1]\r\n2015-03-24 22:10:35,971 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://hnn101-lax1:50030/jobdetails.jsp?jobid=job_201408291723_27582\r\n2015-03-24 22:10:47,051 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 75% complete\r\n2015-03-24 22:10:59,619 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 92% complete\r\n2015-03-24 22:11:15,720 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete\r\n2015-03-24 22:11:15,721 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:\r\n\r\nHadoopVersion PigVersion UserId StartedAt FinishedAt Features\r\n2.0.0-cdh4.4.0 0.11.0-cdh4.4.0 bganapathy 2015-03-24 22:09:46 2015-03-24 22:11:15 LIMIT\r\n\r\nSuccess!\r\n\r\nJob Stats (time in seconds):\r\nJobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MedianMapTime MaxReduceTime MinReduceTime AvgReduceTime MedianReducetime Alias FeatureOutputs\r\njob_201408291723_27581 12 1 8 5 7 7 15 15 15 15 testEs\r\n\r\njob_201408291723_27582 1 1 5 5 5 5 17 17 17 17 testEs ztmp_inventory_tool_sample/invData,\r\n\r\nInput(s):\r\nSuccessfully read 1200000 records (19469816 bytes) from: \"/user/bganapathy/flightPKeys\"\r\n\r\nOutput(s):\r\nSuccessfully stored 100000 records in: \"ztmp_inventory_tool_sample/invData\""},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/comments/87735346","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/405#issuecomment-87735346","issue_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/405","id":87735346,"user":{"login":"costin","id":76245,"avatar_url":"https://avatars.githubusercontent.com/u/76245?v=3","gravatar_id":"","url":"https://api.github.com/users/costin","html_url":"https://github.com/costin","followers_url":"https://api.github.com/users/costin/followers","following_url":"https://api.github.com/users/costin/following{/other_user}","gists_url":"https://api.github.com/users/costin/gists{/gist_id}","starred_url":"https://api.github.com/users/costin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/costin/subscriptions","organizations_url":"https://api.github.com/users/costin/orgs","repos_url":"https://api.github.com/users/costin/repos","events_url":"https://api.github.com/users/costin/events{/privacy}","received_events_url":"https://api.github.com/users/costin/received_events","type":"User","site_admin":false},"created_at":"2015-03-30T16:00:44Z","updated_at":"2015-03-30T16:00:44Z","body":"The versions should work. Pig is somewhat old but still supported. Again there are no errors in the logs and clearly there is an indication of the number of records. Consider turning on logging to see the network traffic and what ES nodes are hit."}}