{"action":"created","issue":{"url":"https://api.github.com/repos/voldemort/voldemort/issues/290","labels_url":"https://api.github.com/repos/voldemort/voldemort/issues/290/labels{/name}","comments_url":"https://api.github.com/repos/voldemort/voldemort/issues/290/comments","events_url":"https://api.github.com/repos/voldemort/voldemort/issues/290/events","html_url":"https://github.com/voldemort/voldemort/issues/290","id":98615850,"number":290,"title":"Data loss after rebalancing","user":{"login":"elad","id":3621643,"avatar_url":"https://avatars.githubusercontent.com/u/3621643?v=3","gravatar_id":"","url":"https://api.github.com/users/elad","html_url":"https://github.com/elad","followers_url":"https://api.github.com/users/elad/followers","following_url":"https://api.github.com/users/elad/following{/other_user}","gists_url":"https://api.github.com/users/elad/gists{/gist_id}","starred_url":"https://api.github.com/users/elad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elad/subscriptions","organizations_url":"https://api.github.com/users/elad/orgs","repos_url":"https://api.github.com/users/elad/repos","events_url":"https://api.github.com/users/elad/events{/privacy}","received_events_url":"https://api.github.com/users/elad/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-08-02T15:08:17Z","updated_at":"2015-08-11T15:43:52Z","closed_at":null,"body":"Hey,\r\n\r\nI'm trying to add a node to a Voldemort cluster (two-node to three-node, all on the same machine) and in the process of rebalancing seem to be losing about a third of the data.\r\n\r\nHere's my process:\r\n\r\n  * Setup a two-node cluster\r\n  * Run a BnP job to load 100 records into store \"foobar\"\r\n  * Verify all of the keys are there - all of the keys are present\r\n  * Add node and rebalance (see below for the steps - this might be where the problem is)\r\n  * Verify all of the keys are there - see about 33% data loss\r\n\r\nHere's the original `cluster.xml`:\r\n\r\n```\r\n<cluster>\r\n  <name>foobar</name>\r\n  <server>\r\n    <id>0</id>\r\n    <host>localhost</host>\r\n    <http-port>8081</http-port>\r\n    <socket-port>6666</socket-port>\r\n    <admin-port>6667</admin-port>\r\n    <partitions>0, 2, 4, 6, 8, 10</partitions>\r\n  </server>\r\n  <server>\r\n    <id>1</id>\r\n    <host>localhost</host>\r\n    <http-port>8085</http-port>\r\n    <socket-port>6670</socket-port>\r\n    <admin-port>6671</admin-port>\r\n    <partitions>1, 3, 5, 7, 9, 11</partitions>\r\n  </server>\r\n</cluster>\r\n```\r\n\r\nI'm following the instructions from http://www.project-voldemort.com/voldemort/rebalance.html and perhaps a better way to demonstrate exactly what I'm doing is with the script I use to do it:\r\n\r\n```shell\r\n#!/bin/bash\r\n\r\nWORKDIR=/tmp/vrb\r\nVOLDEMORT_DIR=voldemort-repo\r\nBOOTSTRAP_URL=tcp://localhost:6666\r\nNEW_CLUSTER_DIR=config/node_2\r\n\r\nrm -rf ${WORKDIR}\r\n\r\n# Get current cluster configuration\r\n${VOLDEMORT_DIR}/bin/vadmin.sh meta get cluster.xml,stores.xml --url ${BOOTSTRAP_URL} --dir ${WORKDIR} --node 0\r\nmv ${WORKDIR}/cluster.xml_0 ${WORKDIR}/cluster.xml\r\nmv ${WORKDIR}/stores.xml_0 ${WORKDIR}/stores.xml\r\n\r\n# Rebalance the cluster (\"on paper\")\r\ncp ${NEW_CLUSTER_DIR}/config/cluster.xml ${WORKDIR}/interim.xml\r\n${VOLDEMORT_DIR}/bin/rebalance-cluster-expansion.sh -c ${WORKDIR}/cluster.xml -s ${WORKDIR}/stores.xml -i ${WORKDIR}/interim.xml -o ${WORKDIR}/output\r\n\r\n# Sample keys for verification\r\n${VOLDEMORT_DIR}/bin/run-class.sh voldemort.tools.KeySamplerCLI --url ${BOOTSTRAP_URL} --out-dir ${WORKDIR}/output/key-samples --records-per-partition 5\r\n${VOLDEMORT_DIR}/bin/run-class.sh voldemort.tools.KeyVersionFetcherCLI --url ${BOOTSTRAP_URL} --in-dir ${WORKDIR}/output/key-samples --out-dir ${WORKDIR}/output/key-version-fetches\r\n\r\n# Inspect async jobs on the cluster\r\n${VOLDEMORT_DIR}/bin/vadmin.sh async-job list --url ${BOOTSTRAP_URL}\r\nread -p 'Deal with async jobs if needed, then press enter to proceed... '\r\n\r\n# Copy the cluster/stores configuration to the new node and start it\r\ncp ${WORKDIR}/output/final-cluster.xml ${NEW_CLUSTER_DIR}/config/cluster.xml\r\ncp ${WORKDIR}/stores.xml ${NEW_CLUSTER_DIR}/config/stores.xml\r\nread -p 'Please start the new node... '\r\n\r\n# Broadcast new cluster configuration to all existing nodes\r\n${VOLDEMORT_DIR}/bin/vadmin.sh meta set cluster.xml=${WORKDIR}/output/final-cluster.xml,stores.xml=${WORKDIR}/stores.xml --url ${BOOTSTRAP_URL} --all-nodes --confirm\r\n\r\n# Rebalance the cluster\r\nread -p 'Press enter to rebalance... '\r\n${VOLDEMORT_DIR}/bin/run-class.sh voldemort.tools.RebalanceControllerCLI --url ${BOOTSTRAP_URL} --final-cluster ${WORKDIR}/output/final-cluster.xml --parallelism 8 --proxy-pause 5\r\n\r\n# TODO: Verification\r\n\r\n# Repair\r\n${VOLDEMORT_DIR}/bin/vadmin.sh cleanup orphaned-data --url ${BOOTSTRAP_URL} --all-nodes --confirm\r\n\r\n# TODO: Reset retention job and bounce the cluster\r\n```\r\n\r\nI run the above script and end with this output for the rebalancing part:\r\n\r\n```\r\n[09:31:36,195 voldemort.client.AbstractStoreClientFactory] INFO Client zone-id [-1] Attempting to get raw store [voldsys$_metadata_version_persistence]  [main]\r\n[09:31:36,206 voldemort.client.AbstractStoreClientFactory] INFO Client zone-id [-1] Attempting to get raw store [voldsys$_store_quotas]  [main]\r\n[09:31:36,273 voldemort.client.protocol.admin.AdminClient] INFO Setting quota.enforcement.enabled for localhost:0 [main]\r\n[09:31:36,340 voldemort.client.protocol.admin.AdminClient] INFO Setting quota.enforcement.enabled for localhost:1 [main]\r\n[09:31:36,413 voldemort.client.protocol.admin.AdminClient] INFO Setting quota.enforcement.enabled for localhost:2 [main]\r\n[09:31:36,474 voldemort.client.rebalance.RebalanceController] INFO Validating rebalance plan state. [main]\r\n[09:31:36,475 voldemort.client.rebalance.RebalanceController] INFO Validating state of deployed cluster to confirm its ready for rebalance. [main]\r\n[09:31:36,477 voldemort.client.protocol.admin.AdminClient] INFO Node 0 (localhost) is ready for rebalance. [main]\r\n[09:31:36,478 voldemort.client.protocol.admin.AdminClient] INFO Node 1 (localhost) is ready for rebalance. [main]\r\n[09:31:36,479 voldemort.client.protocol.admin.AdminClient] INFO Node 2 (localhost) is ready for rebalance. [main]\r\n[09:31:36,484 voldemort.client.rebalance.RebalanceController] INFO Starting to execute rebalance Plan! [main]\r\n[09:31:36,486 voldemort.client.protocol.admin.AdminClient] INFO Setting quota.enforcement.enabled for localhost:0 [main]\r\n[09:31:36,542 voldemort.client.protocol.admin.AdminClient] INFO Setting quota.enforcement.enabled for localhost:1 [main]\r\n[09:31:36,593 voldemort.client.protocol.admin.AdminClient] INFO Setting quota.enforcement.enabled for localhost:2 [main]\r\n[09:31:36,652 voldemort.client.protocol.admin.AdminClient] INFO No quota set for GET_THROUGHPUT of store foobar  [main]\r\n[09:31:36,655 voldemort.client.protocol.admin.AdminClient] INFO No quota set for PUT_THROUGHPUT of store foobar  [main]\r\n[09:31:36,657 voldemort.client.protocol.admin.AdminClient] INFO No quota set for GET_ALL_THROUGHPUT of store foobar  [main]\r\n[09:31:36,659 voldemort.client.protocol.admin.AdminClient] INFO No quota set for DELETE_THROUGHPUT of store foobar  [main]\r\n```\r\n\r\nHere's the interim `cluster.xml`:\r\n\r\n```\r\n<cluster>\r\n  <name>foobar</name>\r\n  <server>\r\n    <id>0</id>\r\n    <host>localhost</host>\r\n    <http-port>8081</http-port>\r\n    <socket-port>6666</socket-port>\r\n    <admin-port>6667</admin-port>\r\n    <partitions>0, 2, 4, 6, 8, 10</partitions>\r\n  </server>\r\n  <server>\r\n    <id>1</id>\r\n    <host>localhost</host>\r\n    <http-port>8085</http-port>\r\n    <socket-port>6670</socket-port>\r\n    <admin-port>6671</admin-port>\r\n    <partitions>1, 3, 5, 7, 9, 11</partitions>\r\n  </server>\r\n  <server>\r\n    <id>2</id>\r\n    <host>localhost</host>\r\n    <http-port>8089</http-port>\r\n    <socket-port>6674</socket-port>\r\n    <admin-port>6675</admin-port>\r\n    <partitions></partitions>\r\n  </server>\r\n</cluster>\r\n```\r\n\r\nAnd here's the final `cluster.xml` that ends up on all nodes:\r\n\r\n```\r\n<cluster>\r\n  <name>foobar</name>\r\n  <server>\r\n    <id>0</id>\r\n    <host>localhost</host>\r\n    <http-port>8081</http-port>\r\n    <socket-port>6666</socket-port>\r\n    <admin-port>6667</admin-port>\r\n    <partitions>0, 2, 4, 6</partitions>\r\n  </server>\r\n  <server>\r\n    <id>1</id>\r\n    <host>localhost</host>\r\n    <http-port>8085</http-port>\r\n    <socket-port>6670</socket-port>\r\n    <admin-port>6671</admin-port>\r\n    <partitions>3, 5, 7, 9</partitions>\r\n  </server>\r\n  <server>\r\n    <id>2</id>\r\n    <host>localhost</host>\r\n    <http-port>8089</http-port>\r\n    <socket-port>6674</socket-port>\r\n    <admin-port>6675</admin-port>\r\n    <partitions>1, 8, 10, 11</partitions>\r\n  </server>\r\n</cluster>\r\n```\r\n\r\nAny ideas why I'm losing data? Is there anything I'm doing wrong - or not doing at all?\r\n\r\nThanks! :)"},"comment":{"url":"https://api.github.com/repos/voldemort/voldemort/issues/comments/129936281","html_url":"https://github.com/voldemort/voldemort/issues/290#issuecomment-129936281","issue_url":"https://api.github.com/repos/voldemort/voldemort/issues/290","id":129936281,"user":{"login":"arunthirupathi","id":6520371,"avatar_url":"https://avatars.githubusercontent.com/u/6520371?v=3","gravatar_id":"","url":"https://api.github.com/users/arunthirupathi","html_url":"https://github.com/arunthirupathi","followers_url":"https://api.github.com/users/arunthirupathi/followers","following_url":"https://api.github.com/users/arunthirupathi/following{/other_user}","gists_url":"https://api.github.com/users/arunthirupathi/gists{/gist_id}","starred_url":"https://api.github.com/users/arunthirupathi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arunthirupathi/subscriptions","organizations_url":"https://api.github.com/users/arunthirupathi/orgs","repos_url":"https://api.github.com/users/arunthirupathi/repos","events_url":"https://api.github.com/users/arunthirupathi/events{/privacy}","received_events_url":"https://api.github.com/users/arunthirupathi/received_events","type":"User","site_admin":false},"created_at":"2015-08-11T15:43:52Z","updated_at":"2015-08-11T15:43:52Z","body":"Yep, Sid is right, I completely forgot about that.\n\nThanks,\nArun.\n\nOn Tue, Aug 11, 2015 at 8:25 AM, singhsiddharth <notifications@github.com>\nwrote:\n\n> I think, now we have an admin API to disable the fetches so there is no\n> need to make the fetcher null. Just use that API to disable the fetches.\n> On Aug 11, 2015 8:22 AM, \"ARUNACHALAM THIRUPATHI\" <\n> notifications@github.com>\n> wrote:\n>\n> > Also make sure that no pushes happen during the rebalance. Typically we\n> > achieve that by setting the fetcher property to null, bouncing the\n> cluster,\n> > rebalancing the cluster, set the property back to the right fetcher and\n> > bounce again.\n> >\n> > Thanks,\n> > Arun.\n> >\n> > On Mon, Aug 10, 2015 at 3:04 PM, Elad Efrat <notifications@github.com>\n> > wrote:\n> >\n> > > @stotch <https://github.com/stotch> - thanks for taking a look at\n> this.\n> > > Your suggestion sounds good, I'll give that a try first thing tomorrow\n> > > morning and follow up.\n> > >\n> > > —\n> > > Reply to this email directly or view it on GitHub\n> > > <\n> > https://github.com/voldemort/voldemort/issues/290#issuecomment-129628753\n> >\n> > > .\n> > >\n> >\n> > —\n> > Reply to this email directly or view it on GitHub\n> > <\n> https://github.com/voldemort/voldemort/issues/290#issuecomment-129928210>\n> > .\n>\n> >\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/voldemort/voldemort/issues/290#issuecomment-129928996>\n> .\n>\n"}}