{"push_id":730173163,"size":12,"distinct_size":12,"ref":"refs/heads/feature/replicator","head":"ee1403e36113c93a86e05723f9b84f7ad8d93a6e","before":"35ae729458e6447cc7e1f9642eb65475b5f5d1ec","commits":[{"sha":"b812822ede94cf30136385ffd60f2d2a96a33fbe","author":{"email":"287dfd97d2274fe201ae7191a244ba25e475e13a@gmail.com","name":"Alexander Shorin"},"message":"Spawn Replication workers\n\nTo archive higher performance through concurrency, Replication\nuses workers for all actual hard job.","distinct":true,"url":"https://api.github.com/repos/aio-libs/aiocouchdb/commits/b812822ede94cf30136385ffd60f2d2a96a33fbe"},{"sha":"4f4d66d63dbfc306352c5e6c4e4d6c8e11de0aed","author":{"email":"287dfd97d2274fe201ae7191a244ba25e475e13a@gmail.com","name":"Alexander Shorin"},"message":"Find missing revs","distinct":true,"url":"https://api.github.com/repos/aio-libs/aiocouchdb/commits/4f4d66d63dbfc306352c5e6c4e4d6c8e11de0aed"},{"sha":"4c20253c28f4829714455238a33f484c81681a3e","author":{"email":"287dfd97d2274fe201ae7191a244ba25e475e13a@gmail.com","name":"Alexander Shorin"},"message":"Fetch and flush documents with missed revisions\n\nThis is the most important part of replication worker: once difference\nbetween source and target was found, we need to fetch them from source\nand store them on target. What could be easy?\n\nFirst, we need to a poll of readers to fetch docs from source\nconcurrently - amount of readers is limited by http_connections task\noption. For each document we may fetch many conflicts, so we need to\nsend updates by small batches instead of accumulating one single large\nlist of all the docs. However, we shouldn't batch documents with the\nattachments and flush them ASAP without intermediate buffers.\n\nDifferences from couch_replicator_worker implementation:\n- Open revisions are fetched all at since API call, not one by one\n- Batch docs processing happens in own loop\n- Batch size limit is based on docs amount in list, not on their JSON\n  byte size","distinct":true,"url":"https://api.github.com/repos/aio-libs/aiocouchdb/commits/4c20253c28f4829714455238a33f484c81681a3e"},{"sha":"dd09f841ff5336198f3b25d4cb3bf477159b0a63","author":{"email":"287dfd97d2274fe201ae7191a244ba25e475e13a@gmail.com","name":"Alexander Shorin"},"message":"Monitor all replication async tasks","distinct":true,"url":"https://api.github.com/repos/aio-libs/aiocouchdb/commits/dd09f841ff5336198f3b25d4cb3bf477159b0a63"},{"sha":"2fc99d7375da0d7f7f0dd114df15dd56daa27083","author":{"email":"287dfd97d2274fe201ae7191a244ba25e475e13a@gmail.com","name":"Alexander Shorin"},"message":"Report about the last received seq\n\nWhen replication is filtered, workers may not process the last\nseq and therfore it cannot be recorded in checkpoint.\n\nThis situation eventually forces to re-replicate old seqs which\ncompromises incremental nature of replication.\n\nThat's why we need ensure that last_seq is recored when changes\nfeed stream ended.","distinct":true,"url":"https://api.github.com/repos/aio-libs/aiocouchdb/commits/2fc99d7375da0d7f7f0dd114df15dd56daa27083"},{"sha":"b3dbfaba5e2a06ee401c4df946f037a44e87b370","author":{"email":"287dfd97d2274fe201ae7191a244ba25e475e13a@gmail.com","name":"Alexander Shorin"},"message":"Be agnostic about the update seq type\n\nhttps://github.com/apache/couchdb/commit/c7ea12fa2","distinct":true,"url":"https://api.github.com/repos/aio-libs/aiocouchdb/commits/b3dbfaba5e2a06ee401c4df946f037a44e87b370"},{"sha":"013d1e5302c56e970b1a816ea55faba7a85f0e24","author":{"email":"287dfd97d2274fe201ae7191a244ba25e475e13a@gmail.com","name":"Alexander Shorin"},"message":"Delegate question of Peer API reliablity to their implementation\n\nReplicator shouldn't know any specifics of underlying protocol\nnor possible communication issues. That's a responsibility of\nthe API interface.\n\nHowever, IPeer interface provides some helpers to retry failed\nrequests with timeout after incremental delay.","distinct":true,"url":"https://api.github.com/repos/aio-libs/aiocouchdb/commits/013d1e5302c56e970b1a816ea55faba7a85f0e24"},{"sha":"357c6b325791c75d58d90b92412905e7a99a5f11","author":{"email":"287dfd97d2274fe201ae7191a244ba25e475e13a@gmail.com","name":"Alexander Shorin"},"message":"Remove knowledge about how changes feeds are works\n\nPrevious implementation used own iteration over the changes feed in\norder to fetch changes. But it makes problematic to restore the feed\nin case of connection failure since exception will be raised on\nfeed.next() yield from. And feed.next() certainly has no idea how to\nrestart the feed - it's not his job.\n\nSo we move that iteration to Peer abstract interface and ask it just\nto put the changes into our queue. How it will keep changes feed alive\nand works with it is not a replicator business since all it need is\nthe data from the feed.","distinct":true,"url":"https://api.github.com/repos/aio-libs/aiocouchdb/commits/357c6b325791c75d58d90b92412905e7a99a5f11"},{"sha":"afb48889e97879581fd7c043cea6455a89c9b9f1","author":{"email":"287dfd97d2274fe201ae7191a244ba25e475e13a@gmail.com","name":"Alexander Shorin"},"message":"Implement ReplicationManager","distinct":true,"url":"https://api.github.com/repos/aio-libs/aiocouchdb/commits/afb48889e97879581fd7c043cea6455a89c9b9f1"},{"sha":"7a1acdec700b80ee3ad61bc35ee9e8daae9b3143","author":{"email":"287dfd97d2274fe201ae7191a244ba25e475e13a@gmail.com","name":"Alexander Shorin"},"message":"Implement replication statistics gathering","distinct":true,"url":"https://api.github.com/repos/aio-libs/aiocouchdb/commits/7a1acdec700b80ee3ad61bc35ee9e8daae9b3143"},{"sha":"2812dfd9839eb255b8500e0dad337ab2dd39966e","author":{"email":"287dfd97d2274fe201ae7191a244ba25e475e13a@gmail.com","name":"Alexander Shorin"},"message":"Count checkpoints that made for stats\n\ncouch_replication provides two metrics on this:\n- couch_replicator.checkpoints.success\n- couch_replicator.checkpoints.failure\n\nHowever, at the same time if checkpoint record fails, replication is\ngoing to be stopped - so there is no much sense to have such metric\nfor now.","distinct":true,"url":"https://api.github.com/repos/aio-libs/aiocouchdb/commits/2812dfd9839eb255b8500e0dad337ab2dd39966e"},{"sha":"ee1403e36113c93a86e05723f9b84f7ad8d93a6e","author":{"email":"287dfd97d2274fe201ae7191a244ba25e475e13a@gmail.com","name":"Alexander Shorin"},"message":"Implement CouchDB 1.x interface for replicator","distinct":true,"url":"https://api.github.com/repos/aio-libs/aiocouchdb/commits/ee1403e36113c93a86e05723f9b84f7ad8d93a6e"}]}