{"push_id":776596901,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"a645a3560ae8743c2bf4459d2855af90198fc3ff","before":"73e25e7454f4d20be3513db3e6357efdc2425638","commits":[{"sha":"a645a3560ae8743c2bf4459d2855af90198fc3ff","author":{"email":"01dbad3a32472a219b04fa2c6a2e42f3815c76f2@gmail.com","name":"Sergey Serebryakov"},"message":"Introduce job prioritization to ExecutionGraph\n\nSee googledoc for details: https://docs.google.com/a/twitter.com/document/d/1BGVAR_AdGXsf08MT0nBLh8s-lYenx7NK7Kk7keD_gwg/edit?usp=sharing\n\nPROBLEM:\nThe jobs are submitted to the worker pool in a sort of insertion order. If the submission order was based on the “size” or “importance” of the job, it could result in less total execution time.\n\nSOLUTION:\nIntroduce a priority queue where ready-to-be-submitted jobs are waiting for the moment when a worker becomes available, being sorted by “importance”. When a job finishes, its unblocked dependees is not submitted to the worker pool, but rather put in the priority queue. When a worker becomes available (i.e. when any job is finished), the most “important” job is taken from the priority queue. There is an additional latency (before, ready jobs are submitted to the worker pool immediately; after, they are all first put in the priority queue, and only then the most prioritized ones are chosen and submitted).\n\nAdditionally, make tracking of ready-to-be-submitted jobs smarter with a counter in status table which is decremented with every finished dependency.\n\nTesting Done:\n$ ./pants test tests/python/pants_test:all\nSUCCESS\n\nCI green: https://travis-ci.org/megaserg/pants/builds/78164959\n\nBugs closed: 2109\n\nReviewed at https://rbcommons.com/s/twitter/r/2601/","distinct":true,"url":"https://api.github.com/repos/pantsbuild/pants/commits/a645a3560ae8743c2bf4459d2855af90198fc3ff"}]}