{"action":"created","issue":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/373","labels_url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/373/labels{/name}","comments_url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/373/comments","events_url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/373/events","html_url":"https://github.com/elasticsearch/elasticsearch-hadoop/issues/373","id":56873518,"number":373,"title":"[Spark]Is there a way to make elasticsearch-hadoop stick to the load-balancer(client), instead of going trying to ping all the data nodes?","user":{"login":"wingchen","id":798321,"avatar_url":"https://avatars.githubusercontent.com/u/798321?v=3","gravatar_id":"","url":"https://api.github.com/users/wingchen","html_url":"https://github.com/wingchen","followers_url":"https://api.github.com/users/wingchen/followers","following_url":"https://api.github.com/users/wingchen/following{/other_user}","gists_url":"https://api.github.com/users/wingchen/gists{/gist_id}","starred_url":"https://api.github.com/users/wingchen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wingchen/subscriptions","organizations_url":"https://api.github.com/users/wingchen/orgs","repos_url":"https://api.github.com/users/wingchen/repos","events_url":"https://api.github.com/users/wingchen/events{/privacy}","received_events_url":"https://api.github.com/users/wingchen/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/labels/feature","name":"feature","color":"84b6eb"},{"url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/labels/question","name":"question","color":"cc317c"},{"url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/labels/rest","name":"rest","color":"DDDDDD"},{"url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/labels/spark","name":"spark","color":"006b75"},{"url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/labels/v2.1.0.Beta4","name":"v2.1.0.Beta4","color":"bfdadc"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-02-06T21:54:48Z","updated_at":"2015-02-09T21:46:32Z","closed_at":null,"body":"This is spark related.\r\n\r\nI got the following exception with the code:\r\n\r\n```\r\nscala> sc.getConf.get(\"es.nodes.discovery\")\r\nres2: String = false\r\n\r\nscala> val rdd = sc.esRDD(\"news/searchable-news-article\")\r\nrdd: org.elasticsearch.spark.rdd.ScalaEsRDD = ScalaEsRDD[1] at RDD at AbstractEsRDD.scala:17\r\n\r\nscala> rdd.count()\r\n```\r\n\r\nException:\r\n\r\n```\r\norg.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot find node with id [gy_92kljRluUFT9N022KDw] (is HTTP enabled?) from shard\r\n...\r\n\tat org.elasticsearch.hadoop.util.Assert.notNull(Assert.java:40)\r\n\tat org.elasticsearch.hadoop.rest.RestRepository.getReadTargetShards(RestRepository.java:277)\r\n\tat org.elasticsearch.hadoop.rest.RestService.findPartitions(RestService.java:241)\r\n\tat org.elasticsearch.spark.rdd.AbstractEsRDD.esPartitions$lzycompute(AbstractEsRDD.scala:49)\r\n\tat org.elasticsearch.spark.rdd.AbstractEsRDD.esPartitions(AbstractEsRDD.scala:48)\r\n\tat org.elasticsearch.spark.rdd.AbstractEsRDD.getPartitions(AbstractEsRDD.scala:25)\r\n\tat org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:206)\r\n\tat org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)\r\n```\r\n\r\nIt seems to go and try other nodes even after `es.nodes.discovery` is explicitly turned off.\r\n\r\nHere comes the `/_nodes?pretty` (I removed some sensitive info):\r\n\r\n```\r\n{\r\n  \"ok\" : true,\r\n  \"cluster_name\" : \"nyuzu\",\r\n  \"nodes\" : {\r\n    \"JRpaF5f7T2uLfJ90-sLBLA\" : {\r\n      \"name\" : \"es-client001.mycom.com\",\r\n      \"transport_address\" : \"inet[/11.222.111.68:9300]\",\r\n      \"hostname\" : \"es-client001.mycom.com\",\r\n      \"version\" : \"0.90.5\",\r\n      \"http_address\" : \"inet[/11.222.111.68:9500]\",\r\n      \"attributes\" : {\r\n        \"data\" : \"false\",\r\n        \"max_local_storage_nodes\" : \"1\",\r\n        \"master\" : \"false\"\r\n      }\r\n    },\r\n    \"Q_5mAcbBS_KGm6dEVp3oqQ\" : {\r\n      \"name\" : \"es-data002.mycom.com\",\r\n      \"transport_address\" : \"inet[/11.333.222.69:9300]\",\r\n      \"hostname\" : \"es-data002.mycom.com\",\r\n      \"version\" : \"0.90.5\",\r\n      \"attributes\" : {\r\n        \"max_local_storage_nodes\" : \"1\",\r\n        \"master\" : \"false\"\r\n      }\r\n    },\r\n    \"4d63O0bTSxyoVLE7DNDuiA\" : {\r\n      \"name\" : \"es-master01.mycom.com\",\r\n      \"transport_address\" : \"inet[/11.221.163.11:9300]\",\r\n      \"hostname\" : \"es-master01.mycom.com\",\r\n      \"version\" : \"0.90.5\",\r\n      \"attributes\" : {\r\n        \"data\" : \"false\",\r\n        \"max_local_storage_nodes\" : \"1\",\r\n        \"master\" : \"true\"\r\n      }\r\n    },\r\n    \"gy_92kljRluUFT9N022KDw\" : {\r\n      \"name\" : \"es-data001.mycom.com\",\r\n      \"transport_address\" : \"inet[/11.113.18.70:9300]\",\r\n      \"hostname\" : \"es-data001.mycom.com\",\r\n      \"version\" : \"0.90.5\",\r\n      \"attributes\" : {\r\n        \"max_local_storage_nodes\" : \"1\",\r\n        \"master\" : \"false\"\r\n      }\r\n    }\r\n  }\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/comments/73597100","html_url":"https://github.com/elasticsearch/elasticsearch-hadoop/issues/373#issuecomment-73597100","issue_url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/373","id":73597100,"user":{"login":"wingchen","id":798321,"avatar_url":"https://avatars.githubusercontent.com/u/798321?v=3","gravatar_id":"","url":"https://api.github.com/users/wingchen","html_url":"https://github.com/wingchen","followers_url":"https://api.github.com/users/wingchen/followers","following_url":"https://api.github.com/users/wingchen/following{/other_user}","gists_url":"https://api.github.com/users/wingchen/gists{/gist_id}","starred_url":"https://api.github.com/users/wingchen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wingchen/subscriptions","organizations_url":"https://api.github.com/users/wingchen/orgs","repos_url":"https://api.github.com/users/wingchen/repos","events_url":"https://api.github.com/users/wingchen/events{/privacy}","received_events_url":"https://api.github.com/users/wingchen/received_events","type":"User","site_admin":false},"created_at":"2015-02-09T21:46:32Z","updated_at":"2015-02-09T21:46:32Z","body":"This is too little for a gist, so I am posting it here:\r\n\r\n```\r\n15/02/09 13:43:21 DEBUG JavaEsRDD: Found client nodes [10.10.10.10:9500]\r\n15/02/09 13:43:21 DEBUG JavaEsRDD: Filtered discovered only nodes [develop.mycom.com:9200] to client-only []\r\n15/02/09 13:43:21 DEBUG JavaEsRDD: Discovered Elasticsearch version [0.90.5]\r\n```\r\nIt turns out that all the other logs are the same, but `JavaEsRDD` did not find any client-only nodes.\r\n\r\nI am looking into why too."}}