{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/619","labels_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/619/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/619/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/619/events","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/619","id":120760847,"number":619,"title":"how to achieve a long live connections using spark api","user":{"login":"jiacai2050","id":3848910,"avatar_url":"https://avatars.githubusercontent.com/u/3848910?v=3","gravatar_id":"","url":"https://api.github.com/users/jiacai2050","html_url":"https://github.com/jiacai2050","followers_url":"https://api.github.com/users/jiacai2050/followers","following_url":"https://api.github.com/users/jiacai2050/following{/other_user}","gists_url":"https://api.github.com/users/jiacai2050/gists{/gist_id}","starred_url":"https://api.github.com/users/jiacai2050/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jiacai2050/subscriptions","organizations_url":"https://api.github.com/users/jiacai2050/orgs","repos_url":"https://api.github.com/users/jiacai2050/repos","events_url":"https://api.github.com/users/jiacai2050/events{/privacy}","received_events_url":"https://api.github.com/users/jiacai2050/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-12-07T12:09:19Z","updated_at":"2015-12-08T17:25:13Z","closed_at":null,"body":"I can write my rdds to es with following java code\r\n```\r\nJavaEsSpark.saveToEs(rdd, indexName);\r\n```\r\nActually I am using Spark Streaming to write data to es constantly.\r\nIn my box, I can see lots of `TIME_WAITED` when execute `netstat -an | grep 9200`.\r\nClearly, every time the `saveToEs` method gets invoked, it creates a new connection to add data to es.\r\n\r\nSo is there any way to achieve a long live connections ?\r\n\r\nFYI, I found [this link](https://groups.google.com/forum/#!topic/elasticsearch/jbguN90ubJg), but still have no idea how to do."},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/comments/162952465","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/619#issuecomment-162952465","issue_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/619","id":162952465,"user":{"login":"costin","id":76245,"avatar_url":"https://avatars.githubusercontent.com/u/76245?v=3","gravatar_id":"","url":"https://api.github.com/users/costin","html_url":"https://github.com/costin","followers_url":"https://api.github.com/users/costin/followers","following_url":"https://api.github.com/users/costin/following{/other_user}","gists_url":"https://api.github.com/users/costin/gists{/gist_id}","starred_url":"https://api.github.com/users/costin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/costin/subscriptions","organizations_url":"https://api.github.com/users/costin/orgs","repos_url":"https://api.github.com/users/costin/repos","events_url":"https://api.github.com/users/costin/events{/privacy}","received_events_url":"https://api.github.com/users/costin/received_events","type":"User","site_admin":false},"created_at":"2015-12-08T17:25:13Z","updated_at":"2015-12-08T17:25:13Z","body":"`saveToEs` starts a Spark task that in turn, looks at the ES index and then writes the data in parallel.\r\nThe `long-lived` connection doesn't really work in this case since this is not just one program where state is passed around but actually a number of jobs that are being fired very rapidly. The `TIME_WAITED` status actually indicates that the connection has closed but it is now waiting on any stray packets that might have not arrived.\r\nThis is typically an OS setting - for a more aggressive setup, you can minimize its value.\r\n\r\nBack to the issue, there's no much the connector can do at this point - it does not have access to the runtime and it is not possible to pass a connection from Spark to ES-Hadoop (where would that connection be by the way)?\r\n\r\nTry to minimize the frequency of `saveToEs` calls by using a bigger/larger RDD. If that is not working for you, than maybe using a different client (maybe the java one) would work better for you instead of es-hadoop.\r\nBasically in that case, you would run everything from within the driver and the whole partitioning of the RDD would not be useful. And everything would run within the same machine since otherwise how would you share your 'connection'..."}}