{"action":"created","issue":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/387","labels_url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/387/labels{/name}","comments_url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/387/comments","events_url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/387/events","html_url":"https://github.com/elasticsearch/elasticsearch-hadoop/issues/387","id":59879822,"number":387,"title":"Does it make sense to see similiar performance with/without 'es.nodes.client.only' set to 'true'  in spark? this related to #373","user":{"login":"wingchen","id":798321,"avatar_url":"https://avatars.githubusercontent.com/u/798321?v=3","gravatar_id":"","url":"https://api.github.com/users/wingchen","html_url":"https://github.com/wingchen","followers_url":"https://api.github.com/users/wingchen/followers","following_url":"https://api.github.com/users/wingchen/following{/other_user}","gists_url":"https://api.github.com/users/wingchen/gists{/gist_id}","starred_url":"https://api.github.com/users/wingchen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wingchen/subscriptions","organizations_url":"https://api.github.com/users/wingchen/orgs","repos_url":"https://api.github.com/users/wingchen/repos","events_url":"https://api.github.com/users/wingchen/events{/privacy}","received_events_url":"https://api.github.com/users/wingchen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-03-04T23:26:51Z","updated_at":"2015-03-09T22:16:19Z","closed_at":null,"body":"Hi @costin,\r\n\r\nThank you again for #373. The fix works great.\r\n\r\nWe have been trying to benchmark the performance with and without `es.nodes.client.only` set to `true`. Here comes the ES cluster settings:\r\n\r\n![mycom com](https://cloud.githubusercontent.com/assets/798321/6495685/2b7d8930-c281-11e4-936f-ff1b89bdfd72.png)\r\n\r\nThere is around 27GB and 2,685,643 documents in this cluster. All of them have http enabled. We then started to use 1 spark worker node to dump entire data out with the following code:\r\n\r\nhttps://gist.github.com/wingchen/24345b5dabf212a19ee6\r\n\r\nWe set/unset `conf.set(\"es.nodes.client.only\", \"true\") ` at line 11 to see the difference.\r\n\r\nIt turns out that the performance is not much different from each other.\r\n\r\nWith `es.nodes.client.only` set to `true`:\r\n\r\n```\r\nElapsed time: 575937176018ns\r\nElapsed time: 472600809446ns\r\nElapsed time: 551257582522ns\r\nElapsed time: 537133446720ns\r\nElapsed time: 586939805228ns\r\nElapsed time: 648629441787ns\r\nElapsed time: 529784087027ns\r\nElapsed time: 526775918447ns\r\n```\r\n\r\nWithout `es.nodes.client.only` set to `true`:\r\n\r\n```\r\nElapsed time: 538304918249ns\r\nElapsed time: 629962009599ns\r\nElapsed time: 525998454515ns\r\nElapsed time: 639182008909ns\r\nElapsed time: 663308006015ns\r\nElapsed time: 680316998433ns\r\nElapsed time: 582389258596ns\r\nElapsed time: 506762219830ns\r\n```\r\n\r\nWe tried this for couple more times with different ways to loop through all the docs and got similar result.\r\n\r\nDoes this make sense to you? Did we do anything wrong?"},"comment":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/comments/77956108","html_url":"https://github.com/elasticsearch/elasticsearch-hadoop/issues/387#issuecomment-77956108","issue_url":"https://api.github.com/repos/elasticsearch/elasticsearch-hadoop/issues/387","id":77956108,"user":{"login":"wingchen","id":798321,"avatar_url":"https://avatars.githubusercontent.com/u/798321?v=3","gravatar_id":"","url":"https://api.github.com/users/wingchen","html_url":"https://github.com/wingchen","followers_url":"https://api.github.com/users/wingchen/followers","following_url":"https://api.github.com/users/wingchen/following{/other_user}","gists_url":"https://api.github.com/users/wingchen/gists{/gist_id}","starred_url":"https://api.github.com/users/wingchen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wingchen/subscriptions","organizations_url":"https://api.github.com/users/wingchen/orgs","repos_url":"https://api.github.com/users/wingchen/repos","events_url":"https://api.github.com/users/wingchen/events{/privacy}","received_events_url":"https://api.github.com/users/wingchen/received_events","type":"User","site_admin":false},"created_at":"2015-03-09T22:16:19Z","updated_at":"2015-03-09T22:16:19Z","body":"@costin \r\nI increased spark workers number to 4 (12 cores) and still see similar results (it's way faster now):\r\n\r\nQueries directly to nodes:\r\nElapsed time: 169175076846ns\r\nElapsed time: 227021836078ns\r\nElapsed time: 247021145438ns\r\nElapsed time: 259099202476ns\r\nElapsed time: 196787504525ns\r\nElapsed time: 232730755253ns\r\nElapsed time: 163212245504ns\r\nElapsed time: 226552320074ns\r\n\r\nQueries to client:\r\nElapsed time: 277207796177ns\r\nElapsed time: 209881209785ns\r\nElapsed time: 272959493646ns\r\nElapsed time: 200681801838ns\r\nElapsed time: 232259698945ns\r\nElapsed time: 164348253066ns\r\nElapsed time: 254794330333ns\r\nElapsed time: 238033701832ns\r\n\r\nThe numbers are too odd, so I went into newrelic to monitor the traffic of two different configurations: client only and otherwise.\r\n\r\nIt turns out that there is no significant amount of traffic going through client at all during the whole time:\r\n![client](https://cloud.githubusercontent.com/assets/798321/6565547/0feacb2e-c66e-11e4-9381-1d48359d4c6a.png)\r\n\r\nHowever, the data nodes are transmitting data without any problem:\r\n![nodes](https://cloud.githubusercontent.com/assets/798321/6565586/6314df60-c66e-11e4-8fb1-6aa297bcc7d2.png)\r\n(Please note that I ran client-only and otherwise interchangeably during the 6 times of execution.)\r\n\r\nIn this case, it seems that no matter I set `es.nodes.client.only` or not, data is passing out through data nodes, not client. Can you confirm it?\r\n\r\nThe ES cluster configuration is here: https://gist.github.com/wingchen/fb332414d0e1e5a3b844 Please let me know if any of the configuration is preventing client nodes configuration from working.\r\n\r\nThanks"}}