{"push_id":649534841,"size":2,"distinct_size":2,"ref":"refs/heads/backtracking-scheduling-dev","head":"4d17effd5c1e514caaa7b5a6a11f6ec639e302f5","before":"9ac127e05f5776f67f6c4625c22c427668bdec39","commits":[{"sha":"0a94333b97b99378e520449e9abcc43a15a10ed7","author":{"email":"180f55e8350155df1048ad760eeaa1cff137f014@apache.org","name":"Maximilian Michels"},"message":"[JobManager] default to single-user mode, save the last ExecutionGraph for resuming","distinct":true,"url":"https://api.github.com/repos/mxm/flink/commits/0a94333b97b99378e520449e9abcc43a15a10ed7"},{"sha":"4d17effd5c1e514caaa7b5a6a11f6ec639e302f5","author":{"email":"180f55e8350155df1048ad760eeaa1cff137f014@apache.org","name":"Maximilian Michels"},"message":"[scheduling] implement backtracking of intermediate results\n\nFor batch programs, we currently schedule all tasks which are sources\nand let them kick off the execution of the connected tasks. This\napproach bears some problems when executing large dataflows with many\nbranches. With backtracking, we traverse the execution graph\noutput-centrically (from the sinks) in a depth-first manner. This\nenables us to use resources differently.\n\nIn the course of backtracking, only tasks will be executed that are\nrequired to supply inputs to the current task. When a job is newly\nsubmitted, this means that the backtracking will reach the\nsources. When the job has been previously executed and intermediate\nresults are available, old ResultPartitions to resume from can be\nrequested while backtracking.\n\nBacktracking is disabled by default. It can be enabled by setting the\nScheduleMode in JobGraph to BACKTRACKING.\n\nCHANGELOG\n- new scheduling mode: backtracking\n- backtracks from the sinks of an ExecutionGraph\n- checks the availability of IntermediatePartitionResults\n- marks ExecutionVertex to be scheduled\n- caches ResultPartitions and reloads them\n- resumes from intermediate results\n- test for general behavior of backtracking (BacktrackingTest)\n- test for resuming from an intermediate result (ResumeITCase)\n- test for releasing of cached ResultPartitions (ResultPartitionManagerTest)\n- allow multiple consumers per blocking intermediate result (batch)","distinct":true,"url":"https://api.github.com/repos/mxm/flink/commits/4d17effd5c1e514caaa7b5a6a11f6ec639e302f5"}]}