{"push_id":827846614,"size":1,"distinct_size":1,"ref":"refs/heads/suguwork","head":"bde6d88fe192e55a9da85471ee1fc0ce0cffe181","before":"3ae5b3b510eeba05e7a24cde8c8fc3694f6e6aab","commits":[{"sha":"bde6d88fe192e55a9da85471ee1fc0ce0cffe181","author":{"email":"0ee711480472e44bb0ec6060537f82d554add06e@google.com","name":"Sugu Sougoumarane"},"message":"tabletserver: fix race condition\n\nWe discovered an interesting race condition between Begin\nand StopServer:\n\n* Begin gets called: reserves a request, but takes time to\nperform the next step, state is SERVING.\n* StopService gets called: transitions state to SHUTTING_DOWN.\n* StopService calls WaitForTx: the pool is empty. So, it\nproceeds to perform other shutdown operations.\n* Begin opens a transaction, because it was given permission.\nThe pool is not empty any more.\n* StopService calls txpool.Close, which gets angry because\nno transactions should be in it.\n\nOne possible solution I considered was to reserve an entry\nin the transaction pool during startRequest. Then Begin would\nlater fill it out with the connection obtained after BEGIN\nsucceeded.  This would eliminate the race, but it also\nintroduces weird dependencies between TabletServer and TxPool.\nIt would also require Begin to un-reserve if the BEGIN failed.\nOverall, I felt the above approach would lead to unreadable code.\n\nSo, I went with the following approach:\n* Introduce a new 'begins' waitgroup that counts number of Begin\ncalls in flight.\n* Increment & Decrement it in startRequest and endRequest.\n* Make those who want to stop query service wait for begins to\nbe zero before waiting on tx pool to go empty.\n\nSo, if there are any Begins in flight when we go into shut down,\nwe wait for them to complete. As they complete, they all reserve\na connection in the tx pool. Once the Begins are done, no new begins\nare allowed, and those that completed all have created transactions.\nSo, at this point, it's safe to wait for tx pool to go empty.\n\nThis is not really testable. So, we have to rely on the above proof\nand verification that the code was written accordingly.","distinct":true,"url":"https://api.github.com/repos/youtube/vitess/commits/bde6d88fe192e55a9da85471ee1fc0ce0cffe181"}]}