{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/9373","labels_url":"https://api.github.com/repos/apache/spark/issues/9373/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/9373/comments","events_url":"https://api.github.com/repos/apache/spark/issues/9373/events","html_url":"https://github.com/apache/spark/pull/9373","id":114210942,"number":9373,"title":"[SPARK-11419][STREAMING] Parallel recovery for FileBasedWriteAheadLog + minor recovery tweaks","user":{"login":"brkyvz","id":5243515,"avatar_url":"https://avatars.githubusercontent.com/u/5243515?v=3","gravatar_id":"","url":"https://api.github.com/users/brkyvz","html_url":"https://github.com/brkyvz","followers_url":"https://api.github.com/users/brkyvz/followers","following_url":"https://api.github.com/users/brkyvz/following{/other_user}","gists_url":"https://api.github.com/users/brkyvz/gists{/gist_id}","starred_url":"https://api.github.com/users/brkyvz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brkyvz/subscriptions","organizations_url":"https://api.github.com/users/brkyvz/orgs","repos_url":"https://api.github.com/users/brkyvz/repos","events_url":"https://api.github.com/users/brkyvz/events{/privacy}","received_events_url":"https://api.github.com/users/brkyvz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":22,"created_at":"2015-10-30T06:19:47Z","updated_at":"2015-11-12T06:23:28Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/9373","html_url":"https://github.com/apache/spark/pull/9373","diff_url":"https://github.com/apache/spark/pull/9373.diff","patch_url":"https://github.com/apache/spark/pull/9373.patch"},"body":"The support for closing WriteAheadLog files after writes was just merged in. Closing every file after a write is a very expensive operation as it creates many small files on S3. It's not necessary to enable it on HDFS anyway.\r\n\r\nHowever, when you have many small files on S3, recovery takes very long. In addition, files start stacking up pretty quickly, and deletes may not be able to keep up, therefore deletes can also be parallelized.\r\n\r\nThis PR adds support for the two parallelization steps mentioned above, in addition to a couple more failures I encountered during recovery. "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/156013324","html_url":"https://github.com/apache/spark/pull/9373#issuecomment-156013324","issue_url":"https://api.github.com/repos/apache/spark/issues/9373","id":156013324,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-11-12T06:23:28Z","updated_at":"2015-11-12T06:23:28Z","body":"**[Test build #45712 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45712/consoleFull)** for PR 9373 at commit [`dbb31e3`](https://github.com/apache/spark/commit/dbb31e372d178c70bb3a6f8b18c931ce0867d4b2)."}}