{"push_id":711076001,"size":1,"distinct_size":1,"ref":"refs/heads/fix-flow-control","head":"4606d29d2129158169cca9514970259e723b5869","before":"23553db838f3f111b39508238740b1b7a28db0c1","commits":[{"sha":"4606d29d2129158169cca9514970259e723b5869","author":{"email":"049a7d046ad9739d3f5690bf40485f6d8ec65510@gmail.com","name":"Peter Inglesby"},"message":"Fix flow control\n\nMorph can push messages onto a queue faster than Angler can pull them\noff the queue, and RabbitMQ's own back-pressure mechanisms don't seem to\nwork, so we have tried to implement our own flow control mechanism.\nHowever, the first attempt didn't work.\n\nThe first attempt involved the producer working out how fast the\nconsumer was consuming the queue, and then sleeping between producing\neach record, so as to not produce messages faster than the consumer was\nconsuming them.  This was stupid and I should have my licence to\ncomputer revoked.  The problem is that the rate that messages are\nconsumed is either limited by a bottleneck on the consumer (in which\ncase what we were doing makes sense) or is limited by the rate that\nmessages are produced (cos it can't, y'know, consume messages that\naren't being produced).  So if the producer hits a slow patch, the\nproducer will see that the consumer is also consuming slowly, so it'll\nsubsequently limit itself so that it doesn't overwhelm the consumer,\nwhich is actually fine and not at all overwhelmed.\n\nThis commit should fix that.  Every 1000 records, the producer checks to\nsee whether the bot_record_consumer queue has grown above a threshold.\nIf the queue is too big, it sleeps for a random amount of time, before\nchecking again.  There may be several other producers, so the random\nlength of the sleep is to allow each producer to get a chance at being\nable to continue producing once the queue size drops below the\nthreshold.","distinct":true,"url":"https://api.github.com/repos/sebbacon/morph/commits/4606d29d2129158169cca9514970259e723b5869"}]}