{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/606","labels_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/606/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/606/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/606/events","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/606","id":116460132,"number":606,"title":"Hive loading data into ES error: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed","user":{"login":"vanjor","id":864664,"avatar_url":"https://avatars.githubusercontent.com/u/864664?v=3","gravatar_id":"","url":"https://api.github.com/users/vanjor","html_url":"https://github.com/vanjor","followers_url":"https://api.github.com/users/vanjor/followers","following_url":"https://api.github.com/users/vanjor/following{/other_user}","gists_url":"https://api.github.com/users/vanjor/gists{/gist_id}","starred_url":"https://api.github.com/users/vanjor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanjor/subscriptions","organizations_url":"https://api.github.com/users/vanjor/orgs","repos_url":"https://api.github.com/users/vanjor/repos","events_url":"https://api.github.com/users/vanjor/events{/privacy}","received_events_url":"https://api.github.com/users/vanjor/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/hive","name":"hive","color":"02d7e1"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-11-12T01:30:52Z","updated_at":"2015-11-12T14:26:09Z","closed_at":null,"body":"Thanks for @costin advice , I create a individual issue \r\n\r\nUsing hive 1.2.1 loading data from hive to ES,\r\nI was succeeded to load billions of data into ES by using hive, but when I try to update the data into ES, the job will failed after several hours.\r\nBriefly, It okay to loading data into an empty index by using hive, but it will failed in the midway when updating those large scale ES data by using hive.\r\n\r\n@costin, I read your another blog: https://discuss.elastic.co/t/spark-es-batch-write-retry-count-negative-value-is-ignored/25436/2\r\n```\r\nes.batch.write.retry.count should work. Note that the connector has two types of retries:,\r\n```\r\nI have no idea which one is the type I encountered, network hiccups or document reject. And is that If i set it to a negative number , will avoid  the job stop midway? \r\n\r\nAnd the same condition, full update the index is failed, but recreate the index is okay, it that update index cost more ES resources?\r\n\r\n\r\nFollow with detail errors:\r\n-----------------------------------\r\nHadoop job throws error, and stopped:\r\n\r\n org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[10.1.23.134:9200, es.op.koudai.com:9200, 10.1.23.132:9200, 10.1.23.131:9200, 10.1.23.130:9200, 10.1.23.133:9200]] \r\n\r\n```\r\n2015-11-06 15:40:55,230 ERROR [main] org.elasticsearch.hadoop.rest.NetworkClient: Node [Read timed out] failed (10.1.23.133:9200); no other nodes left - aborting...\r\n2015-11-06 15:40:55,259 FATAL [main] org.apache.hadoop.hive.ql.exec.mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {\"user_id\":\"492923825\",\"is_register\":null,\"register_time\":null\"}\r\n\tat org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:518)\r\n\tat org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)\r\n\tat org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)\r\n\tat org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)\r\n\tat org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)\r\n\tat org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)\r\n\tat java.security.AccessController.doPrivileged(Native Method)\r\n\tat javax.security.auth.Subject.doAs(Subject.java:415)\r\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)\r\n\tat org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)\r\nCaused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[10.1.23.134:9200, es.op.koudai.com:9200, 10.1.23.132:9200, 10.1.23.131:9200, 10.1.23.130:9200, 10.1.23.133:9200]] \r\n\tat org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:142)\r\n\tat org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:317)\r\n\tat org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:313)\r\n\tat org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:150)\r\n\tat org.elasticsearch.hadoop.rest.RestRepository.tryFlush(RestRepository.java:209)\r\n\tat org.elasticsearch.hadoop.rest.RestRepository.flush(RestRepository.java:232)\r\n\tat org.elasticsearch.hadoop.rest.RestRepository.doWriteToIndex(RestRepository.java:185)\r\n\tat org.elasticsearch.hadoop.rest.RestRepository.writeProcessedToIndex(RestRepository.java:164)\r\n\tat org.elasticsearch.hadoop.hive.EsHiveOutputFormat$EsHiveRecordWriter.write(EsHiveOutputFormat.java:63)\r\n\tat org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:753)\r\n\tat org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)\r\n\tat org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)\r\n\tat org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)\r\n\tat org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)\r\n\tat org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:162)\r\n\tat org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508)\r\n\t... 9 more\r\n\r\n2015-11-06 15:40:55,259 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: 3 finished. closing... \r\n2015-11-06 15:40:55,259 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: RECORDS_IN:879999\r\n2015-11-06 15:40:55,259 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:0\r\n```\r\nmy mapping config is \r\n```sql\r\nCREATE EXTERNAL TABLE es.buyer_es (\r\n  user_id  string,\r\n  is_register int,\r\n  register_time  string,\r\n  xxx\r\n)\r\nSTORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'\r\nTBLPROPERTIES('es.resource' = 'xxx/buyers','es.nodes'= 'xxx',\r\n              'es.port'= '9200','es.mapping.id' = 'user_id','es.index.auto.create' = 'true',\r\n              'es.batch.size.entries' = '1000','es.batch.write.retry.count' = '10000','es.batch.write.retry.wait' = '10s',\r\n              'es.batch.write.refresh' = 'false','es.nodes.discovery' = 'true','es.nodes.client.only' = 'false'\r\n             );\r\n```\r\n\r\n\r\nmy insert&update script is \r\n\r\n```\r\nINSERT OVERWRITE TABLE es.buyer_es \r\nselect * from xxx\r\n```"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/comments/156115334","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/606#issuecomment-156115334","issue_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/606","id":156115334,"user":{"login":"vanjor","id":864664,"avatar_url":"https://avatars.githubusercontent.com/u/864664?v=3","gravatar_id":"","url":"https://api.github.com/users/vanjor","html_url":"https://github.com/vanjor","followers_url":"https://api.github.com/users/vanjor/followers","following_url":"https://api.github.com/users/vanjor/following{/other_user}","gists_url":"https://api.github.com/users/vanjor/gists{/gist_id}","starred_url":"https://api.github.com/users/vanjor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanjor/subscriptions","organizations_url":"https://api.github.com/users/vanjor/orgs","repos_url":"https://api.github.com/users/vanjor/repos","events_url":"https://api.github.com/users/vanjor/events{/privacy}","received_events_url":"https://api.github.com/users/vanjor/received_events","type":"User","site_admin":false},"created_at":"2015-11-12T14:26:09Z","updated_at":"2015-11-12T14:26:09Z","body":"Thanks for your reply:)\r\n1. Shall ES maintains the traffic control mechanism ? I thought it was okay to using  'es.batch.write.retry.count' = '10000','es.batch.write.retry.wait' = '10s',  in case of ES is busy, client will auto slower down request for the repeat waiting during retry and by this config, it should take 10000*10s = 27 hours for job to fail, but my job failed after 4 hours. why?\r\n\r\n2.  I am runing the experiment by defaults config to update 10+ billions data, 115 tasks on 5 ES nodes with 16 core CPU. waiting the result now.\r\nmy ES version:1.7.1, ES-Hadoop version:2.1.1"}}