{"push_id":656161961,"size":1,"distinct_size":1,"ref":"refs/heads/release-3.7","head":"579186aeba940e3ec73093c48e17b5f6f94910d0","before":"2a4f346fe57fb21330857b7eb75153dc8abc4def","commits":[{"sha":"579186aeba940e3ec73093c48e17b5f6f94910d0","author":{"email":"3fb5363a31297d930f52de0d8d2a80bcaae2c58e@redhat.com","name":"Susant Palai"},"message":"rebalance: Introducing local crawl and parallel migration\n\nThe current patch address two part of the design proposed.\n1. Rebalance multiple files in parallel\n2. Crawl only bricks that belong to the current node\n\nBrief design explanation for the above two points.\n\n1. Rebalance multiple files in parallel:\n   -------------------------------------\nThe existing rebalance engine is single threaded. Hence, introduced\nmultiple threads which will be running parallel to the crawler. The\ncurrent rebalance migration is converted to a \"Producer-Consumer\"\nframe work.\n\nWhere Producer is : Crawler\n      Consumer is : Migrating Threads\n\nCrawler: Crawler is the main thread. The job of the crawler is now\nlimited to fix-layout of each directory and add the files which are\neligible for the migration to a global queue in a round robin manner\nso that we will use all the disk resources efficiently. Hence, the\ncrawler will not be \"blocked\" by migration process.\n\nProducer: Producer will monitor the global queue. If any file is\nadded to this queue, it will dqueue that entry and migrate the file.\nCurrently 20 migration threads are spawned at the beginning of the\nrebalance process. Hence, multiple file migration happens in parallel.\n\n2. Crawl only bricks that belong to the current node:\n   --------------------------------------------------\nAs rebalance process is spawned per node, it migrates only the files\nthat belongs to it's own node for the sake of load balancing. But it\nalso reads entries from the whole cluster, which is not necessary as\nreaddir hits other nodes.\n\nNew Design:\n        As part of the new design the rebalancer decides the subvols\nthat are local to the rebalancer node by checking the node-uuid of\nroot directory prior to the crawler starts. Hence, readdir won't hit\nthe whole cluster  as it has already the context of local subvols and\nalso node-uuid request for each file can be avoided. This makes the\nrebalance process \"more scalable\".\n\nChange-Id: I6f1b44086a09df8ca23935fd213509c70cc0c050\nBUG: 1217381\nSigned-off-by: Susant Palai <spalai@redhat.com>\nReviewed-on: http://review.gluster.org/10466\nTested-by: Gluster Build System <jenkins@build.gluster.com>\nTested-by: NetBSD Build System\nReviewed-by: N Balachandran <nbalacha@redhat.com>","distinct":true,"url":"https://api.github.com/repos/gluster/glusterfs/commits/579186aeba940e3ec73093c48e17b5f6f94910d0"}]}