{"action":"created","issue":{"url":"https://api.github.com/repos/hpcugent/hanythingondemand/issues/129","labels_url":"https://api.github.com/repos/hpcugent/hanythingondemand/issues/129/labels{/name}","comments_url":"https://api.github.com/repos/hpcugent/hanythingondemand/issues/129/comments","events_url":"https://api.github.com/repos/hpcugent/hanythingondemand/issues/129/events","html_url":"https://github.com/hpcugent/hanythingondemand/issues/129","id":115830270,"number":129,"title":"Performance tune Spark","user":{"login":"ehiggs","id":28823,"avatar_url":"https://avatars.githubusercontent.com/u/28823?v=3","gravatar_id":"","url":"https://api.github.com/users/ehiggs","html_url":"https://github.com/ehiggs","followers_url":"https://api.github.com/users/ehiggs/followers","following_url":"https://api.github.com/users/ehiggs/following{/other_user}","gists_url":"https://api.github.com/users/ehiggs/gists{/gist_id}","starred_url":"https://api.github.com/users/ehiggs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ehiggs/subscriptions","organizations_url":"https://api.github.com/users/ehiggs/orgs","repos_url":"https://api.github.com/users/ehiggs/repos","events_url":"https://api.github.com/users/ehiggs/events{/privacy}","received_events_url":"https://api.github.com/users/ehiggs/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-11-09T08:30:32Z","updated_at":"2015-12-03T10:44:05Z","closed_at":null,"body":"By default, Spark uses something silly like only 500MB per executor and only three executors. We should add a Spark autogen script to calculate how many executors there should be and how much memory they should get."},"comment":{"url":"https://api.github.com/repos/hpcugent/hanythingondemand/issues/comments/161593894","html_url":"https://github.com/hpcugent/hanythingondemand/issues/129#issuecomment-161593894","issue_url":"https://api.github.com/repos/hpcugent/hanythingondemand/issues/129","id":161593894,"user":{"login":"ehiggs","id":28823,"avatar_url":"https://avatars.githubusercontent.com/u/28823?v=3","gravatar_id":"","url":"https://api.github.com/users/ehiggs","html_url":"https://github.com/ehiggs","followers_url":"https://api.github.com/users/ehiggs/followers","following_url":"https://api.github.com/users/ehiggs/following{/other_user}","gists_url":"https://api.github.com/users/ehiggs/gists{/gist_id}","starred_url":"https://api.github.com/users/ehiggs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ehiggs/subscriptions","organizations_url":"https://api.github.com/users/ehiggs/orgs","repos_url":"https://api.github.com/users/ehiggs/repos","events_url":"https://api.github.com/users/ehiggs/events{/privacy}","received_events_url":"https://api.github.com/users/ehiggs/received_events","type":"User","site_admin":false},"created_at":"2015-12-03T10:44:05Z","updated_at":"2015-12-03T10:44:05Z","body":"#131 has what appears to be correct parameters for Spark, and in the logs, I see that Spark is asking for the appropriate resources, but for some reason Yarn is not releasing them. There are two issues people are running into:\r\n\r\n1. We ask for `spark.executor.instances 2` on a single node, but Yarn only grants one container. If we are on multiple nodes and ask for 5, then we only get 2 (one per node). This limits the cores and memory in use.\r\n\r\n2. Some users in a classroom setting have their settings ignored completely and end up with the default Spark memory settings. i.e. \r\n\r\n```\r\nspark.executor.cores 8\r\nspark.executor.instances 2\r\nspark.executor.memory 18432M\r\n``` \r\n\r\nResults in 1 executor  with 512M."}}