{"push_id":863939301,"size":1,"distinct_size":1,"ref":"refs/heads/branch-1.6","head":"199e4cb2113707862b146c5cfb4c6820adc1f309","before":"05666e09bdceafe25540e674efdd6eb70fe37fa0","commits":[{"sha":"199e4cb2113707862b146c5cfb4c6820adc1f309","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-11419][STREAMING] Parallel recovery for FileBasedWriteAheadLog + minor recovery tweaks\n\nThe support for closing WriteAheadLog files after writes was just merged in. Closing every file after a write is a very expensive operation as it creates many small files on S3. It's not necessary to enable it on HDFS anyway.\n\nHowever, when you have many small files on S3, recovery takes very long. In addition, files start stacking up pretty quickly, and deletes may not be able to keep up, therefore deletes can also be parallelized.\n\nThis PR adds support for the two parallelization steps mentioned above, in addition to a couple more failures I encountered during recovery.\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #9373 from brkyvz/par-recovery.\n\n(cherry picked from commit 7786f9cc0790d27854a1e184f66a9b4df4d040a2)\nSigned-off-by: Tathagata Das <tathagata.das1565@gmail.com>","distinct":true,"url":"https://api.github.com/repos/apache/spark/commits/199e4cb2113707862b146c5cfb4c6820adc1f309"}]}