{"action":"created","issue":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/346","labels_url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/346/labels{/name}","comments_url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/346/comments","events_url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/346/events","html_url":"https://github.com/elasticsearch/elasticsearch-hadoop/issues/346","id":52893429,"number":346,"title":"TaskAttemptId string is not properly formed","user":{"login":"ujlim","id":10303021,"avatar_url":"https://avatars.githubusercontent.com/u/10303021?v=3","gravatar_id":"","url":"https://api.github.com/users/ujlim","html_url":"https://github.com/ujlim","followers_url":"https://api.github.com/users/ujlim/followers","following_url":"https://api.github.com/users/ujlim/following{/other_user}","gists_url":"https://api.github.com/users/ujlim/gists{/gist_id}","starred_url":"https://api.github.com/users/ujlim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ujlim/subscriptions","organizations_url":"https://api.github.com/users/ujlim/orgs","repos_url":"https://api.github.com/users/ujlim/repos","events_url":"https://api.github.com/users/ujlim/events{/privacy}","received_events_url":"https://api.github.com/users/ujlim/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/labels/enhancement","name":"enhancement","color":"fbca04"},{"url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/labels/hive","name":"hive","color":"02d7e1"},{"url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/labels/v2.0.3","name":"v2.0.3","color":"bfdadc"},{"url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/labels/v2.1.0.Beta4","name":"v2.1.0.Beta4","color":"bfdadc"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2014-12-26T10:10:55Z","updated_at":"2015-01-07T09:27:57Z","closed_at":null,"body":"I've done INSERT-TO-ES with Hadoop 2.4.0 and es-hadoop 2.0.2 or 2.1.0.Beta3, ES 1.4.1.\r\n```sql\r\nCREATE EXTERNAL TABLE es_201409_test (\r\n    mailguid          STRING,\r\n    mailtextcontent   STRING,\r\n    mailtimereceived  STRING,\r\n    mailtimesent      STRING\r\n)\r\nSTORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'\r\nTBLPROPERTIES('es.resource' = '201409/test','es.nodes' = '10.0.53.26','es.port' = '9201', 'es.query' = '?q=*', 'es.mapping.id' = 'mailguid');\r\n\r\nINSERT OVERWRITE TABLE es_201409_test \r\nselect guid as mailguid,\r\n       portdesc as mailtextcontent,\r\n       rcvtime  as mailtimereceived,\r\n       rcvtime  as mailtimesent       \r\nfrom \t\r\nism_tb_ods_XXXX limit 10;\r\n```\r\nBut I've met \"TaskAttemptId string: task1419585839976_0003_r_000000_2 is not properly formed\".\r\nI guess that as like \"task_1419567106175_0509_m_000000\" is right.\r\n\r\nHow can I do to solve this problem? \r\n\r\n\r\n\r\n\r\n ---------------------- Running Log -----------------------------------\r\n```\r\nContainer released by application, AttemptID:attempt_1419585839976_0003_1_00_000000_2 Info:Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {\"key\":{},\"value\":{\"_col0\":\"7F9D9782-B248-E411-9628-001517EB0FA5\",\"_col1\":\"*.google.com\",\"_col2\":\"2014-10-01 00:00:33.0\",\"_col3\":\"2014-10-01 00:00:33.0\"}}\r\n        at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:344)\r\n        at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:223)\r\n        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:164)\r\n        at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)\r\n        at org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)\r\n        at java.security.AccessController.doPrivileged(Native Method)\r\n        at javax.security.auth.Subject.doAs(Subject.java:415)\r\n        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)\r\n        at org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)\r\nCaused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {\"key\":{},\"value\":{\"_col0\":\"7F9D9782-B248-E411-9628-001517EB0FA5\",\"_col1\":\"*.google.com\",\"_col2\":\"2014-10-01 00:00:33.0\",\"_col3\":\"2014-10-01 00:00:33.0\"}}\r\n        at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:329)\r\n        ... 8 more\r\nCaused by: java.lang.IllegalArgumentException: TaskAttemptId string : task1419585839976_0003_r_000000_2 is not properly formed\r\n        at org.apache.hadoop.mapreduce.TaskAttemptID.forName(TaskAttemptID.java:201)\r\n        at org.apache.hadoop.mapred.TaskAttemptID.forName(TaskAttemptID.java:129)\r\n        at org.elasticsearch.hadoop.mr.HadoopCfgUtils.getTaskID(HadoopCfgUtils.java:148)\r\n        at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.detectCurrentInstance(EsOutputFormat.java:263)\r\n        at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:164)\r\n        at org.elasticsearch.hadoop.hive.EsHiveOutputFormat$EsHiveRecordWriter.write(EsHiveOutputFormat.java:58)\r\n        at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:621)\r\n        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)\r\n        at org.apache.hadoop.hive.ql.exec.LimitOperator.processOp(LimitOperator.java:51)\r\n        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)\r\n        at org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)\r\n        at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:320)\r\n        ... 8 more\r\n\r\nContainer released by application, AttemptID:attempt_1419585839976_0003_1_00_000000_3 Info:Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {\"key\":{},\"value\":{\"_col0\":\"7F9D9782-B248-E411-9628-001517EB0FA5\",\"_col1\":\"*.google.com\",\"_col2\":\"2014-10-01 00:00:33.0\",\"_col3\":\"2014-10-01 00:00:33.0\"}}\r\n        at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:344)\r\n        at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:223)\r\n        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:164)\r\n        at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:307)\r\n        at org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:562)\r\n        at java.security.AccessController.doPrivileged(Native Method)\r\n        at javax.security.auth.Subject.doAs(Subject.java:415)\r\n        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)\r\n        at org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:551)\r\nCaused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {\"key\":{},\"value\":{\"_col0\":\"7F9D9782-B248-E411-9628-001517EB0FA5\",\"_col1\":\"*.google.com\",\"_col2\":\"2014-10-01 00:00:33.0\",\"_col3\":\"2014-10-01 00:00:33.0\"}}\r\n        at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:329)\r\n        ... 8 more\r\nCaused by: java.lang.IllegalArgumentException: TaskAttemptId string : task1419585839976_0003_r_000000_3 is not properly formed\r\n        at org.apache.hadoop.mapreduce.TaskAttemptID.forName(TaskAttemptID.java:201)\r\n        at org.apache.hadoop.mapred.TaskAttemptID.forName(TaskAttemptID.java:129)\r\n        at org.elasticsearch.hadoop.mr.HadoopCfgUtils.getTaskID(HadoopCfgUtils.java:148)\r\n        at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.detectCurrentInstance(EsOutputFormat.java:263)\r\n        at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:164)\r\n        at org.elasticsearch.hadoop.hive.EsHiveOutputFormat$EsHiveRecordWriter.write(EsHiveOutputFormat.java:58)\r\n        at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:621)\r\n        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)\r\n        at org.apache.hadoop.hive.ql.exec.LimitOperator.processOp(LimitOperator.java:51)\r\n        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)\r\n        at org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)\r\n        at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:320)\r\n        ... 8 more\r\n], Vertex failed as one or more tasks failed. failedTasks:1]\r\n2014-12-26 18:55:10,273 ERROR tez.TezJobMonitor (SessionState.java:printError(545)) - DAG failed due to vertex failure. failedVertices:1 killedVertices:0\r\n2014-12-26 18:55:10,274 INFO  log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=TezRunDag start=1419587687599 end=1419587710274 duration=22675 from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>\r\n2014-12-26 18:55:10,278 INFO  exec.Task (TezTask.java:execute(179)) - org.apache.tez.common.counters.DAGCounter:\r\n2014-12-26 18:55:10,278 INFO  exec.Task (TezTask.java:execute(181)) -    NUM_FAILED_TASKS: 4\r\n2014-12-26 18:55:10,278 INFO  exec.Task (TezTask.java:execute(181)) -    TOTAL_LAUNCHED_TASKS: 5\r\n2014-12-26 18:55:10,278 INFO  exec.Task (TezTask.java:execute(181)) -    DATA_LOCAL_TASKS: 1\r\n2014-12-26 18:55:10,279 INFO  exec.Task (TezTask.java:execute(179)) - File System Counters:\r\n2014-12-26 18:55:10,279 INFO  exec.Task (TezTask.java:execute(181)) -    FILE: BYTES_READ: 32\r\n2014-12-26 18:55:10,279 INFO  exec.Task (TezTask.java:execute(181)) -    FILE: BYTES_WRITTEN: 399\r\n2014-12-26 18:55:10,279 INFO  exec.Task (TezTask.java:execute(181)) -    FILE: READ_OPS: 0\r\n2014-12-26 18:55:10,279 INFO  exec.Task (TezTask.java:execute(181)) -    FILE: LARGE_READ_OPS: 0\r\n2014-12-26 18:55:10,279 INFO  exec.Task (TezTask.java:execute(181)) -    FILE: WRITE_OPS: 0\r\n2014-12-26 18:55:10,280 INFO  exec.Task (TezTask.java:execute(181)) -    HDFS: BYTES_READ: 131072\r\n2014-12-26 18:55:10,280 INFO  exec.Task (TezTask.java:execute(181)) -    HDFS: BYTES_WRITTEN: 0\r\n2014-12-26 18:55:10,280 INFO  exec.Task (TezTask.java:execute(181)) -    HDFS: READ_OPS: 1\r\n2014-12-26 18:55:10,280 INFO  exec.Task (TezTask.java:execute(181)) -    HDFS: LARGE_READ_OPS: 0\r\n2014-12-26 18:55:10,280 INFO  exec.Task (TezTask.java:execute(181)) -    HDFS: WRITE_OPS: 0\r\n2014-12-26 18:55:10,280 INFO  exec.Task (TezTask.java:execute(179)) - org.apache.tez.common.counters.TaskCounter:\r\n2014-12-26 18:55:10,281 INFO  exec.Task (TezTask.java:execute(181)) -    SPILLED_RECORDS: 10\r\n2014-12-26 18:55:10,281 INFO  exec.Task (TezTask.java:execute(181)) -    GC_TIME_MILLIS: 0\r\n2014-12-26 18:55:10,281 INFO  exec.Task (TezTask.java:execute(181)) -    CPU_MILLISECONDS: -1030\r\n2014-12-26 18:55:10,281 INFO  exec.Task (TezTask.java:execute(181)) -    PHYSICAL_MEMORY_BYTES: 1358917632\r\n2014-12-26 18:55:10,281 INFO  exec.Task (TezTask.java:execute(181)) -    VIRTUAL_MEMORY_BYTES: 2759393280\r\n2014-12-26 18:55:10,282 INFO  exec.Task (TezTask.java:execute(181)) -    COMMITTED_HEAP_BYTES: 1852833792\r\n2014-12-26 18:55:10,282 INFO  exec.Task (TezTask.java:execute(181)) -    INPUT_RECORDS_PROCESSED: 15\r\n2014-12-26 18:55:10,282 INFO  exec.Task (TezTask.java:execute(181)) -    OUTPUT_RECORDS: 10\r\n2014-12-26 18:55:10,282 INFO  exec.Task (TezTask.java:execute(181)) -    OUTPUT_BYTES: 1115\r\n2014-12-26 18:55:10,282 INFO  exec.Task (TezTask.java:execute(181)) -    OUTPUT_BYTES_WITH_OVERHEAD: 1137\r\n2014-12-26 18:55:10,283 INFO  exec.Task (TezTask.java:execute(181)) -    OUTPUT_BYTES_PHYSICAL: 367\r\n2014-12-26 18:55:10,283 INFO  exec.Task (TezTask.java:execute(181)) -    ADDITIONAL_SPILLS_BYTES_WRITTEN: 0\r\n2014-12-26 18:55:10,283 INFO  exec.Task (TezTask.java:execute(181)) -    ADDITIONAL_SPILLS_BYTES_READ: 0\r\n2014-12-26 18:55:10,283 INFO  exec.Task (TezTask.java:execute(181)) -    ADDITIONAL_SPILL_COUNT: 0\r\n2014-12-26 18:55:10,283 INFO  exec.Task (TezTask.java:execute(179)) - org.apache.hadoop.hive.ql.exec.MapOperator$Counter:\r\n2014-12-26 18:55:10,284 INFO  exec.Task (TezTask.java:execute(181)) -    DESERIALIZE_ERRORS: 0\r\n2014-12-26 18:55:10,288 ERROR ql.Driver (SessionState.java:printError(545)) - FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask\r\n2014-12-26 18:55:10,288 INFO  log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=Driver.execute start=1419587682970 end=1419587710288 duration=27318 from=org.apache.hadoop.hive.ql.Driver>\r\n2014-12-26 18:55:10,288 INFO  log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>\r\n2014-12-26 18:55:10,289 INFO  log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=releaseLocks start=1419587710288 end=1419587710289 duration=1 from=org.apache.hadoop.hive.ql.Driver>\r\n2014-12-26 18:55:10,289 ERROR operation.Operation (SQLOperation.java:run(202)) - Error running hive query: \r\norg.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask\r\n        at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:146)\r\n        at org.apache.hive.service.cli.operation.SQLOperation.access$000(SQLOperation.java:68)\r\n        at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:199)\r\n        at java.security.AccessController.doPrivileged(Native Method)\r\n        at javax.security.auth.Subject.doAs(Subject.java:415)\r\n        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)\r\n        at org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:493)\r\n        at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208)\r\n        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\r\n        at java.util.concurrent.FutureTask.run(FutureTask.java:262)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n        at java.lang.Thread.run(Thread.java:745)\r\n2014-12-26 18:55:10,296 INFO  log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>\r\n2014-12-26 18:55:10,297 INFO  log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=releaseLocks start=1419587710296 end=1419587710297 duration=1 from=org.apache.hadoop.hive.ql.Driver>\r\n2014-12-26 18:55:10,297 INFO  tez.TezSessionPoolManager (TezSessionPoolManager.java:close(172)) - Closing tez session default? false\r\n2014-12-26 18:55:10,297 INFO  tez.TezSessionState (TezSessionState.java:close(240)) - Closing Tez Session\r\n2014-12-26 18:55:10,298 INFO  client.TezSession (TezSession.java:stop(257)) - Shutting down Tez Session, sessionName=HIVE-4c86a474-9029-48e6-8403-b4ae94d7d3de, applicationId=application_1419585839976_0003\r\n2014-12-26 18:57:06,967 INFO  exec.TableScanOperator (Operator.java:close(574)) - 0 finished. closing... \r\n2014-12-26 18:57:06,967 INFO  exec.SelectOperator (Operator.java:close(574)) - 1 finished. closing... \r\n2014-12-26 18:57:06,967 INFO  exec.ListSinkOperator (Operator.java:close(574)) - 3 finished. closing... \r\n2014-12-26 18:57:06,968 INFO  exec.ListSinkOperator (Operator.java:close(591)) - 3 Close done\r\n2014-12-26 18:57:06,968 INFO  exec.SelectOperator (Operator.java:close(591)) - 1 Close done\r\n2014-12-26 18:57:06,968 INFO  exec.TableScanOperator (Operator.java:close(591)) - 0 Close done\r\n2014-12-26 18:57:06,978 INFO  log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>\r\n2014-12-26 18:57:06,979 INFO  log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=releaseLocks start=1419587826978 end=1419587826979 duration=1 from=org.apache.hadoop.hive.ql.Driver>\r\n```"},"comment":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/comments/68997326","html_url":"https://github.com/elasticsearch/elasticsearch-hadoop/issues/346#issuecomment-68997326","issue_url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/346","id":68997326,"user":{"login":"ujlim","id":10303021,"avatar_url":"https://avatars.githubusercontent.com/u/10303021?v=3","gravatar_id":"","url":"https://api.github.com/users/ujlim","html_url":"https://github.com/ujlim","followers_url":"https://api.github.com/users/ujlim/followers","following_url":"https://api.github.com/users/ujlim/following{/other_user}","gists_url":"https://api.github.com/users/ujlim/gists{/gist_id}","starred_url":"https://api.github.com/users/ujlim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ujlim/subscriptions","organizations_url":"https://api.github.com/users/ujlim/orgs","repos_url":"https://api.github.com/users/ujlim/repos","events_url":"https://api.github.com/users/ujlim/events{/privacy}","received_events_url":"https://api.github.com/users/ujlim/received_events","type":"User","site_admin":false},"created_at":"2015-01-07T09:27:56Z","updated_at":"2015-01-07T09:27:56Z","body":"I need sometime to do other things before trying your fix. But I'll go as soon as possible!! Thank you."}}