{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/243","labels_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/243/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/243/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/243/events","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/243","id":40109078,"number":243,"title":"\"Cannot find node with id\" exception even when the node is alive and cluster is green.","user":{"login":"searchwizard","id":6089654,"avatar_url":"https://avatars.githubusercontent.com/u/6089654?v=3","gravatar_id":"","url":"https://api.github.com/users/searchwizard","html_url":"https://github.com/searchwizard","followers_url":"https://api.github.com/users/searchwizard/followers","following_url":"https://api.github.com/users/searchwizard/following{/other_user}","gists_url":"https://api.github.com/users/searchwizard/gists{/gist_id}","starred_url":"https://api.github.com/users/searchwizard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/searchwizard/subscriptions","organizations_url":"https://api.github.com/users/searchwizard/orgs","repos_url":"https://api.github.com/users/searchwizard/repos","events_url":"https://api.github.com/users/searchwizard/events{/privacy}","received_events_url":"https://api.github.com/users/searchwizard/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/rest","name":"rest","color":"DDDDDD"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":27,"created_at":"2014-08-12T23:15:26Z","updated_at":"2015-03-23T22:27:33Z","closed_at":null,"body":"I am getting the following exception when pushing data from hadoop M/R job. When this happens, the node in question is responding and cluster is also healthy (green). Also, plenty of resources on the box. CPU usage is less than 30%, free memory is over 50G. With this exception, the hadoop map task is failing and getting restarted and eventually succeeding (may be by connecting to a different ES node). These errors are not consistent. They are very intermittent.\r\n```java\r\norg.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot find node with id [Q4pQkOIJSSi2oXRXGUVs8w]\r\n\tat org.elasticsearch.hadoop.util.Assert.notNull(Assert.java:40)\r\n\tat org.elasticsearch.hadoop.rest.RestRepository.getWriteTargetPrimaryShards(RestRepository.java:251)\r\n\tat org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.initSingleIndex(EsOutputFormat.java:218)\r\n\tat org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:201)\r\n\tat org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.write(EsOutputFormat.java:159)\r\n\tat org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:638)\r\n\tat org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)\r\n\tat afi.search.hadoop.es.ESMapper1.map(ESMapper1.java:227)\r\n\tat afi.search.hadoop.es.ESMapper1.map(ESMapper1.java:1)\r\n\tat org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)\r\n\tat org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)\r\n\tat org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)\r\n\tat org.apache.hadoop.mapred.Child$4.run(Child.java:255)\r\n\tat java.security.AccessController.doPrivileged(Native Method)\r\n\tat javax.security.auth.Subject.doAs(Subject.java:415)\r\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)\r\n\tat org.apache.hadoop.mapred.Child.main(Child.java:249)\r\n```"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/comments/85230969","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/243#issuecomment-85230969","issue_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/243","id":85230969,"user":{"login":"searchwizard","id":6089654,"avatar_url":"https://avatars.githubusercontent.com/u/6089654?v=3","gravatar_id":"","url":"https://api.github.com/users/searchwizard","html_url":"https://github.com/searchwizard","followers_url":"https://api.github.com/users/searchwizard/followers","following_url":"https://api.github.com/users/searchwizard/following{/other_user}","gists_url":"https://api.github.com/users/searchwizard/gists{/gist_id}","starred_url":"https://api.github.com/users/searchwizard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/searchwizard/subscriptions","organizations_url":"https://api.github.com/users/searchwizard/orgs","repos_url":"https://api.github.com/users/searchwizard/repos","events_url":"https://api.github.com/users/searchwizard/events{/privacy}","received_events_url":"https://api.github.com/users/searchwizard/received_events","type":"User","site_admin":false},"created_at":"2015-03-23T22:27:33Z","updated_at":"2015-03-23T22:27:33Z","body":"I am not sure if the settings helped! When I originally reported the issue, my settings are same as you reported."}}