{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4051","labels_url":"https://api.github.com/repos/apache/spark/issues/4051/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4051/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4051/events","html_url":"https://github.com/apache/spark/pull/4051","id":54379252,"number":4051,"title":"SPARK-4585. Spark dynamic executor allocation should use minExecutors as...","user":{"login":"sryza","id":654855,"avatar_url":"https://avatars.githubusercontent.com/u/654855?v=3","gravatar_id":"","url":"https://api.github.com/users/sryza","html_url":"https://github.com/sryza","followers_url":"https://api.github.com/users/sryza/followers","following_url":"https://api.github.com/users/sryza/following{/other_user}","gists_url":"https://api.github.com/users/sryza/gists{/gist_id}","starred_url":"https://api.github.com/users/sryza/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sryza/subscriptions","organizations_url":"https://api.github.com/users/sryza/orgs","repos_url":"https://api.github.com/users/sryza/repos","events_url":"https://api.github.com/users/sryza/events{/privacy}","received_events_url":"https://api.github.com/users/sryza/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2015-01-14T21:30:07Z","updated_at":"2015-01-26T02:05:35Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4051","html_url":"https://github.com/apache/spark/pull/4051","diff_url":"https://github.com/apache/spark/pull/4051.diff","patch_url":"https://github.com/apache/spark/pull/4051.patch"},"body":"... initial number"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/71406518","html_url":"https://github.com/apache/spark/pull/4051#issuecomment-71406518","issue_url":"https://api.github.com/repos/apache/spark/issues/4051","id":71406518,"user":{"login":"sryza","id":654855,"avatar_url":"https://avatars.githubusercontent.com/u/654855?v=3","gravatar_id":"","url":"https://api.github.com/users/sryza","html_url":"https://github.com/sryza","followers_url":"https://api.github.com/users/sryza/followers","following_url":"https://api.github.com/users/sryza/following{/other_user}","gists_url":"https://api.github.com/users/sryza/gists{/gist_id}","starred_url":"https://api.github.com/users/sryza/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sryza/subscriptions","organizations_url":"https://api.github.com/users/sryza/orgs","repos_url":"https://api.github.com/users/sryza/repos","events_url":"https://api.github.com/users/sryza/events{/privacy}","received_events_url":"https://api.github.com/users/sryza/received_events","type":"User","site_admin":false},"created_at":"2015-01-26T02:05:35Z","updated_at":"2015-01-26T02:05:35Z","body":"> @sryza Is the point of not requiring these configs that the users don't really know how many executors they actually want?\r\n\r\nExactly.  From my perspective, one of the main motivations for dynamic allocation in the first place was removing the need to set `--num-executors`.  People have long complained about the long list configurations users need to know for MapReduce, but at this point Spark actually requires more to get decent resource utilization.  This particular JIRA was filed Hive didn't know how to set the max executors in a general way.  What specific advice would you give to a user about how to set this property?\r\n\r\n> Also defaulting the max to Int.MaxValue doesn't make much sense in my opinion because the whole point of doing this is to share resources in the cluster.\r\n\r\nThis is the job of the cluster resource manager.  Users and apps shouldn't need self-regulate their usage. YARN will throttle resources when there is contention.  The main responsibility of the app is just to not ask for / sit on more resources than it can take advantage of, which dynamic allocation helps with in general, and which SPARK-4136 will help enforce more strictly."}}