{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8065","labels_url":"https://api.github.com/repos/apache/spark/issues/8065/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8065/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8065/events","html_url":"https://github.com/apache/spark/pull/8065","id":99994425,"number":8065,"title":"[SPARK-9396] [yarn] Spark yarn allocator does not call \"removeContainerRequest\" for allocated Container requests, resulting in bloated ask[] toYarn RM.","user":{"login":"prakhar-jauhari","id":13618614,"avatar_url":"https://avatars.githubusercontent.com/u/13618614?v=3","gravatar_id":"","url":"https://api.github.com/users/prakhar-jauhari","html_url":"https://github.com/prakhar-jauhari","followers_url":"https://api.github.com/users/prakhar-jauhari/followers","following_url":"https://api.github.com/users/prakhar-jauhari/following{/other_user}","gists_url":"https://api.github.com/users/prakhar-jauhari/gists{/gist_id}","starred_url":"https://api.github.com/users/prakhar-jauhari/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prakhar-jauhari/subscriptions","organizations_url":"https://api.github.com/users/prakhar-jauhari/orgs","repos_url":"https://api.github.com/users/prakhar-jauhari/repos","events_url":"https://api.github.com/users/prakhar-jauhari/events{/privacy}","received_events_url":"https://api.github.com/users/prakhar-jauhari/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-08-10T07:35:48Z","updated_at":"2015-08-10T07:40:16Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8065","html_url":"https://github.com/apache/spark/pull/8065","diff_url":"https://github.com/apache/spark/pull/8065.diff","patch_url":"https://github.com/apache/spark/pull/8065.patch"},"body":"Spark-1.2 : Spark's AM does not call removeConatinerRequest for fulfilled container request.\r\n\r\nAs long as the cluster size is large enough to allocate the bloated container requests, containers are sent to spark yarn allocator in allocate response, spark yarn allocator uses missing number of container to launch new executors and release the extra allocated containers.\r\n\r\nThe problem increase in case of a long running job with large executor memory requirements. In this case when ever a executor gets killed, the next ask to yarn Resource manager (RM) is of n+1 containers, which might be served by the RM if it still has enough resources, else RM starts reserving cluster resources for a containers which are not even required by spark in the first place. This causes inefficient resource utilization of cluster resources.\r\n\r\nI have added and tested changes for removing fulfilled container requests in spark 1.2 code. @vanzin can you please review the changes. Thanks in advance.\r\n\r\nThe contribution is original work and that I license the work to the project under the project's open source license. \r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/129338206","html_url":"https://github.com/apache/spark/pull/8065#issuecomment-129338206","issue_url":"https://api.github.com/repos/apache/spark/issues/8065","id":129338206,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-08-10T07:40:15Z","updated_at":"2015-08-10T07:40:15Z","body":"Can one of the admins verify this patch?"}}