{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/731","labels_url":"https://api.github.com/repos/apache/spark/issues/731/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/731/comments","events_url":"https://api.github.com/repos/apache/spark/issues/731/events","html_url":"https://github.com/apache/spark/pull/731","id":33263425,"number":731,"title":"SPARK-1706: Allow multiple executors per worker in Standalone mode","user":{"login":"CodingCat","id":678008,"avatar_url":"https://avatars.githubusercontent.com/u/678008?v=3","gravatar_id":"","url":"https://api.github.com/users/CodingCat","html_url":"https://github.com/CodingCat","followers_url":"https://api.github.com/users/CodingCat/followers","following_url":"https://api.github.com/users/CodingCat/following{/other_user}","gists_url":"https://api.github.com/users/CodingCat/gists{/gist_id}","starred_url":"https://api.github.com/users/CodingCat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CodingCat/subscriptions","organizations_url":"https://api.github.com/users/CodingCat/orgs","repos_url":"https://api.github.com/users/CodingCat/repos","events_url":"https://api.github.com/users/CodingCat/events{/privacy}","received_events_url":"https://api.github.com/users/CodingCat/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":109,"created_at":"2014-05-11T15:02:34Z","updated_at":"2015-03-05T19:12:13Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/731","html_url":"https://github.com/apache/spark/pull/731","diff_url":"https://github.com/apache/spark/pull/731.diff","patch_url":"https://github.com/apache/spark/pull/731.patch"},"body":"resubmit of https://github.com/apache/spark/pull/636  for a totally different algorithm\r\n\r\nhttps://issues.apache.org/jira/browse/SPARK-1706\r\n\r\nIn current implementation, the user has to start multiple workers in a server for starting multiple executors in a server, which introduces additional overhead due to the more JVM processes...\r\n\r\nIn this patch, I changed the scheduling logic in master to enable the user to start multiple executor processes within the same JVM process.\r\n\r\n1. user configure spark.executor.maxCoreNumPerExecutor to suggest the maximum core he/she would like to allocate to each executor\r\n\r\n2. Master assigns the executors to the workers with the major consideration on the memoryPerExecutor and the worker.freeMemory, and tries to allocate as many as possible cores to the executor ```min(min(memoryPerExecutor, worker.freeCore), maxLeftCoreToAssign)``` where ```maxLeftCoreToAssign = maxExecutorCanAssign * maxCoreNumPerExecutor```\r\n\r\n---------------------------------------\r\n\r\nOther small changes include\r\n\r\nchange memoryPerSlave in ApplicationDescription to memoryPerExecutor, as \"Slave\" is overrided to represent both worker and executor in the documents... (we have some discussion on this before?)"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/77430389","html_url":"https://github.com/apache/spark/pull/731#issuecomment-77430389","issue_url":"https://api.github.com/repos/apache/spark/issues/731","id":77430389,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-03-05T19:12:13Z","updated_at":"2015-03-05T19:12:13Z","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28302/\nTest PASSed."}}