{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/570","labels_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/570/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/570/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/570/events","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/570","id":110918544,"number":570,"title":"Spark throwing an exception as \"org.elasticsearch.hadoop.rest.EsHadoopTransportException: java.net.BindException\"","user":{"login":"Prashant-Pal","id":6836305,"avatar_url":"https://avatars.githubusercontent.com/u/6836305?v=3","gravatar_id":"","url":"https://api.github.com/users/Prashant-Pal","html_url":"https://github.com/Prashant-Pal","followers_url":"https://api.github.com/users/Prashant-Pal/followers","following_url":"https://api.github.com/users/Prashant-Pal/following{/other_user}","gists_url":"https://api.github.com/users/Prashant-Pal/gists{/gist_id}","starred_url":"https://api.github.com/users/Prashant-Pal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Prashant-Pal/subscriptions","organizations_url":"https://api.github.com/users/Prashant-Pal/orgs","repos_url":"https://api.github.com/users/Prashant-Pal/repos","events_url":"https://api.github.com/users/Prashant-Pal/events{/privacy}","received_events_url":"https://api.github.com/users/Prashant-Pal/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/question","name":"question","color":"cc317c"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/spark","name":"spark","color":"006b75"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/v2.1.2","name":"v2.1.2","color":"bfdadc"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/v2.2.0-m2","name":"v2.2.0-m2","color":"bfdadc"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-10-12T07:04:56Z","updated_at":"2015-10-12T13:02:25Z","closed_at":"2015-10-12T13:02:25Z","body":"Hi Team,\r\n\r\nI am using elasticsearch-hadoop with the spark for events processing.\r\n\r\nSo when we are processing the events with 3 evt per second I am able to do the same successfully but when bandwidth is increased to 5 times that is 15 events per second then i am getting the below exception. \r\n\r\n15/10/12 10:42:15 ERROR JobScheduler: Error running job streaming job 1444626720000 ms.1\r\norg.elasticsearch.hadoop.rest.EsHadoopTransportException: java.net.BindException: Address already in use\r\n        at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:121)\r\n        at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:317)\r\n        at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:313)\r\n        at org.elasticsearch.hadoop.rest.RestClient.count(RestClient.java:392)\r\n        at org.elasticsearch.hadoop.rest.RestRepository.count(RestRepository.java:456)\r\n        at org.elasticsearch.spark.rdd.AbstractEsRDD.count(AbstractEsRDD.scala:49)\r\n\r\n\r\nThis seems the issue like sockets are not getting closed after the operation so what is the param or how we can close the sockets.\r\n Below is the API I am using for search : \r\nJavaEsSpark.esRDD(sc, \"json_events/events\").count();\r\n\r\n\r\nLet me know if anyone has any info for the same.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/comments/147392138","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/570#issuecomment-147392138","issue_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/570","id":147392138,"user":{"login":"costin","id":76245,"avatar_url":"https://avatars.githubusercontent.com/u/76245?v=3","gravatar_id":"","url":"https://api.github.com/users/costin","html_url":"https://github.com/costin","followers_url":"https://api.github.com/users/costin/followers","following_url":"https://api.github.com/users/costin/following{/other_user}","gists_url":"https://api.github.com/users/costin/gists{/gist_id}","starred_url":"https://api.github.com/users/costin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/costin/subscriptions","organizations_url":"https://api.github.com/users/costin/orgs","repos_url":"https://api.github.com/users/costin/repos","events_url":"https://api.github.com/users/costin/events{/privacy}","received_events_url":"https://api.github.com/users/costin/received_events","type":"User","site_admin":false},"created_at":"2015-10-12T13:02:25Z","updated_at":"2015-10-12T13:02:25Z","body":"Calling count 600/s is just  a complete waste of resources. I assume you are trying to double check whether the data went in but you already know this if POST request has succeeded.\r\nStepping aside for a bit, calling a service (whether it's Elastic or something else) 600 per second for a counter is simply way too ... unoptimal. \r\nUse your own counter instead and use that instead - that is instead of calling count 600/s, call it every 2s or 5s and expect it to be 1200 or 3000 items more.\r\nBatches are your friend.\r\n\r\nThe documentation on REST and the API in Elasticsearch can be easily found in the docs - no need to reproduce them here. And yes, you do have to do parsing of the data. \r\nHowever the [cat api](https://www.elastic.co/guide/en/elasticsearch/reference/master/cat-count.html) might be useful as it is meant for the command-line.\r\n\r\nLast but not least, turning on logging implies configuring log4j which means specifying a parameter within its configuration file. Again, there's plenty of documentation out there including in Spark itself.\r\n\r\nI'm going to close the issue as the underlying bug (discovered through this issue) was fixed and your request can't be addressed by the project. For further discussions, please see the [forum](discuss.elastic.co).\r\nI maintain my recommendation to rethink a bit your block of code since a write is acknowledged once it is being performed, there's no need to verify it again. And writing multiple events/data at once is better for all parties then writing it one by one. The bigger the batch, the better the performance.\r\n\r\n"}}