{"push_id":543848858,"size":1,"distinct_size":1,"ref":"refs/heads/branch-1.2","head":"056149d10c64900fa15ed01843bf952ceb5708ea","before":"f04ff9d37ae312dccd16843df7d0e3a03c30e403","commits":[{"sha":"056149d10c64900fa15ed01843bf952ceb5708ea","author":{"email":"5d14e010a5b345b784fd604efcc6cebb79f97468@gmail.com","name":"zsxwing"},"message":"[SPARK-4951][Core] Fix the issue that a busy executor may be killed\n\nA few changes to fix this issue:\n\n1. Handle the case that receiving `SparkListenerTaskStart` before `SparkListenerBlockManagerAdded`.\n2. Don't add `executorId` to `removeTimes` when the executor is busy.\n3. Use `HashMap.retain` to safely traverse the HashMap and remove items.\n4. Use the same lock in ExecutorAllocationManager and ExecutorAllocationListener to fix the race condition in `totalPendingTasks`.\n5. Move the blocking codes out of the message processing code in YarnSchedulerActor.\n\nAuthor: zsxwing <zsxwing@gmail.com>\n\nCloses #3783 from zsxwing/SPARK-4951 and squashes the following commits:\n\nd51fa0d [zsxwing] Add comments\n2e365ce [zsxwing] Remove expired executors from 'removeTimes' and add idle executors back when a new executor joins\n49f61a9 [zsxwing] Eliminate duplicate executor registered warnings\nd4c4e9a [zsxwing] Minor fixes for the code style\n05f6238 [zsxwing] Move the blocking codes out of the message processing code\n105ba3a [zsxwing] Fix the race condition in totalPendingTasks\nd5c615d [zsxwing] Fix the issue that a busy executor may be killed\n\n(cherry picked from commit 6942b974adad396cba2799eac1fa90448cea4da7)\nSigned-off-by: Andrew Or <andrew@databricks.com>","distinct":true,"url":"https://api.github.com/repos/mapr/spark/commits/056149d10c64900fa15ed01843bf952ceb5708ea"}]}