{"action":"created","issue":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/3960","labels_url":"https://api.github.com/repos/influxdb/influxdb/issues/3960/labels{/name}","comments_url":"https://api.github.com/repos/influxdb/influxdb/issues/3960/comments","events_url":"https://api.github.com/repos/influxdb/influxdb/issues/3960/events","html_url":"https://github.com/influxdb/influxdb/issues/3960","id":104583370,"number":3960,"title":"[0.9.4-nightly] possible \"catch up\" bug with nodes down in a cluster","user":{"login":"iansmith","id":230209,"avatar_url":"https://avatars.githubusercontent.com/u/230209?v=3","gravatar_id":"","url":"https://api.github.com/users/iansmith","html_url":"https://github.com/iansmith","followers_url":"https://api.github.com/users/iansmith/followers","following_url":"https://api.github.com/users/iansmith/following{/other_user}","gists_url":"https://api.github.com/users/iansmith/gists{/gist_id}","starred_url":"https://api.github.com/users/iansmith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iansmith/subscriptions","organizations_url":"https://api.github.com/users/iansmith/orgs","repos_url":"https://api.github.com/users/iansmith/repos","events_url":"https://api.github.com/users/iansmith/events{/privacy}","received_events_url":"https://api.github.com/users/iansmith/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/influxdb/influxdb/labels/category/clustering","name":"category/clustering","color":"d111e2"},{"url":"https://api.github.com/repos/influxdb/influxdb/labels/category/writes","name":"category/writes","color":"bfdadc"}],"state":"open","locked":false,"assignee":{"login":"jwilder","id":219935,"avatar_url":"https://avatars.githubusercontent.com/u/219935?v=3","gravatar_id":"","url":"https://api.github.com/users/jwilder","html_url":"https://github.com/jwilder","followers_url":"https://api.github.com/users/jwilder/followers","following_url":"https://api.github.com/users/jwilder/following{/other_user}","gists_url":"https://api.github.com/users/jwilder/gists{/gist_id}","starred_url":"https://api.github.com/users/jwilder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jwilder/subscriptions","organizations_url":"https://api.github.com/users/jwilder/orgs","repos_url":"https://api.github.com/users/jwilder/repos","events_url":"https://api.github.com/users/jwilder/events{/privacy}","received_events_url":"https://api.github.com/users/jwilder/received_events","type":"User","site_admin":false},"milestone":null,"comments":7,"created_at":"2015-09-02T22:18:22Z","updated_at":"2015-09-04T16:51:28Z","closed_at":null,"body":"[I'm pretty new to influx, so please close if this is silly.]\r\n\r\nI'm running a cluster of 3 nodes, per the instructions in the docs on how to create a cluster, on Ubuntu 14.10:\r\n```\r\n2015/09/01 23:10:13 InfluxDB starting, version 0.9.4-nightly-d6cb017, branch master, commit d6cb01722b2f618af84cb9633bd1fb2efb668a8d\r\n```\r\n\r\nThese are all amazon ec2 instances, connected via a security group.  I wanted to see if a writes to working nodes A or C in the cluster while node B was down would eventually get \"corrected\" when B returns.  Can I run a query after B is brought back into the cluster and get the same answers from A, B, or C?\r\n\r\nSo, I used this script to write 10K points, carefully avoiding one of the nodes because I took it down immediately after database creation (I wasn't sure if db creation would affect anything):\r\n```\r\n\r\n#!/bin/bash\r\nset -e\r\n\r\n\r\ndb=\"tempdb7\"\r\ncurl -G http://ip-172-31-13-16.us-west-2.compute.internal:8086/query --data-urlencode \"q=CREATE DATABASE $db\"\r\n\r\n\r\n###\r\n### use this if you want to sleep a bit to give yourself time to take down the node you don't want\r\necho \"sleeping 15 secs so you can turn off  ip-172-31-13-15.us-west-2.compute.internal\"\r\nsleep 15\r\n\r\nfor i in {1..3333}; do\r\n\tt=`expr 1438387200 + $i`\r\n\ttemp1=`echo $((RANDOM % 60))`\r\n\ttemp2=`echo $((RANDOM % 60))`\r\n\ttemp3=`echo $((RANDOM % 60))`\r\n\tcurl -i -XPOST \"http://ip-172-31-13-16.us-west-2.compute.internal:8086/write?db=$db&precision=s\" --data-binary \"cpu_temp,host=server-a,region=us-east value=$temp1 $t\"\r\n\r\n\t#this can be dropped to experiment with writes during the period when a node is down \r\n\t#curl -i -XPOST \"http://ip-172-31-13-15.us-west-2.compute.internal:8086/write?db=$db&precision=s\" --data-binary \"cpu_temp,host=server-b,region=us-west value=$temp2 $t\"\r\n\r\n\t#this one is what should be used if you drop the one above\r\n\tcurl -i -XPOST \"http://ip-172-31-13-14.us-west-2.compute.internal:8086/write?db=$db&precision=s\" --data-binary \"cpu_temp,host=server-b,region=us-west value=$temp2 $t\"\r\n\r\n\tcurl -i -XPOST \"http://ip-172-31-13-14.us-west-2.compute.internal:8086/write?db=$db&precision=s\" --data-binary \"cpu_temp,host=server-c,region=us-west value=$temp3 $t\"\r\n\r\n\techo \"wrote point $i\"\r\ndone\r\n```\r\n\r\nThen a brought the node back up after this script had finished. \r\n\r\nIn the log of the leader, I saw this when I brought up the node that had been previously down:\r\n```\r\n[handoff] 2015/09/02 21:37:07 6666 queued writes sent to node 2 in 25.509379966s\r\n```\r\n\r\nwhich seemed good to me.\r\n\r\nThen I used the script below to read back the data I had just written, using all three of the nodes as sources.  The two that had been up throughout worked fine.  The one that had been down during the writes gave weird values, suggesting that it had about 4700 values out of the expected 10,000.\r\n\r\n```\r\n#!/bin/bash\r\n\r\nset -e\r\n\r\ndb=\"tempdb7\"\r\n\r\nonethirdCount=\"3333\"\r\ntwothirdsCount=\"6666\"\r\n\r\ntestslicebyserver() {\r\n\r\n\tinfluxhost=$1\r\n\tdatahost=$2\r\n\r\n\tshouldbecount=`curl -s -S -G \"http://$influxhost:8086/query\" --data-urlencode \"db=$db\" --data-urlencode \"q=SELECT count(value) FROM cpu_temp WHERE host='$datahost'\" | jq -r .results[0].series[0].columns[1]`\r\n\tn=`curl -s -S -G \"http://$influxhost:8086/query\" --data-urlencode \"db=$db\" --data-urlencode \"q=SELECT count(value) FROM cpu_temp WHERE host='$datahost'\" | jq .results[0].series[0].values[0][1]`\r\n\r\n\tif [ \"$shouldbecount\" != \"count\" ]; then\r\n\t\techo \"expected count but got $shouldbecount\"\r\n#\t\texit 1\r\n\tfi\r\n\r\n\tif [ \"$n\" != \"$onethirdCount\" ]; then\r\n\t\techo \"expected $onethirdCount but got $n\"\r\n\t\techo \"host was $influxhost and data was $datahost\"\r\n#\t\texit 1\r\n\tfi\r\n}\r\n\r\ninfluxnodes=( \"ip-172-31-13-16.us-west-2.compute.internal\" \"ip-172-31-13-15.us-west-2.compute.internal\" \"ip-172-31-13-14.us-west-2.compute.internal\")\r\nservers=(\"server-a\" \"server-b\" \"server-c\")\r\n\r\nfor i in ${influxnodes[@]}; do\r\n\r\n\techo  \"#### influxnode $i\"\r\n\r\n\tfor j in ${servers[@]}; do \r\n\t\ttestslicebyserver $i $j\r\n\tdone\r\n\r\n\techo \"### two thirds test $i\"\r\n\t###do a test of the other field that is not evenly divided\r\n\tn=`curl -s -S -G \"http://$i:8086/query\" --data-urlencode \"db=$db\" --data-urlencode \"q=SELECT count(value) FROM cpu_temp WHERE region='us-west'\"  | jq '.results[0].series[0].values[0][1]'`\r\n\r\n\r\n\tif [ \"$n\" != \"$twothirdsCount\" ]; then\r\n\t\techo \"failed uneven test, expected $twothirdsCount but got $n\"\r\n\t\texit 1\r\n\tfi\r\ndone\r\n```\r\n\r\nPerhaps I'm just confused about how this is supposed to work? That seems the most likely explanation.\r\nthanks\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/comments/137789120","html_url":"https://github.com/influxdb/influxdb/issues/3960#issuecomment-137789120","issue_url":"https://api.github.com/repos/influxdb/influxdb/issues/3960","id":137789120,"user":{"login":"jwilder","id":219935,"avatar_url":"https://avatars.githubusercontent.com/u/219935?v=3","gravatar_id":"","url":"https://api.github.com/users/jwilder","html_url":"https://github.com/jwilder","followers_url":"https://api.github.com/users/jwilder/followers","following_url":"https://api.github.com/users/jwilder/following{/other_user}","gists_url":"https://api.github.com/users/jwilder/gists{/gist_id}","starred_url":"https://api.github.com/users/jwilder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jwilder/subscriptions","organizations_url":"https://api.github.com/users/jwilder/orgs","repos_url":"https://api.github.com/users/jwilder/repos","events_url":"https://api.github.com/users/jwilder/events{/privacy}","received_events_url":"https://api.github.com/users/jwilder/received_events","type":"User","site_admin":false},"created_at":"2015-09-04T16:51:28Z","updated_at":"2015-09-04T16:51:28Z","body":"I'm able to reproduce some issues and am looking into it."}}