{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/701","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/701/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/701/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/701/events","html_url":"https://github.com/scylladb/scylla/issues/701","id":123687737,"number":701,"title":"Iterative add node / decomission node does not stream all the data","user":{"login":"slivne","id":3465480,"avatar_url":"https://avatars.githubusercontent.com/u/3465480?v=3","gravatar_id":"","url":"https://api.github.com/users/slivne","html_url":"https://github.com/slivne","followers_url":"https://api.github.com/users/slivne/followers","following_url":"https://api.github.com/users/slivne/following{/other_user}","gists_url":"https://api.github.com/users/slivne/gists{/gist_id}","starred_url":"https://api.github.com/users/slivne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slivne/subscriptions","organizations_url":"https://api.github.com/users/slivne/orgs","repos_url":"https://api.github.com/users/slivne/repos","events_url":"https://api.github.com/users/slivne/events{/privacy}","received_events_url":"https://api.github.com/users/slivne/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420"},{"url":"https://api.github.com/repos/scylladb/scylla/labels/showstopper","name":"showstopper","color":"e11d21"}],"state":"open","locked":false,"assignee":{"login":"asias","id":320076,"avatar_url":"https://avatars.githubusercontent.com/u/320076?v=3","gravatar_id":"","url":"https://api.github.com/users/asias","html_url":"https://github.com/asias","followers_url":"https://api.github.com/users/asias/followers","following_url":"https://api.github.com/users/asias/following{/other_user}","gists_url":"https://api.github.com/users/asias/gists{/gist_id}","starred_url":"https://api.github.com/users/asias/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/asias/subscriptions","organizations_url":"https://api.github.com/users/asias/orgs","repos_url":"https://api.github.com/users/asias/repos","events_url":"https://api.github.com/users/asias/events{/privacy}","received_events_url":"https://api.github.com/users/asias/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/1","html_url":"https://github.com/scylladb/scylla/milestones/GA","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/1/labels","id":1164307,"number":1,"title":"GA","description":null,"creator":{"login":"dorlaor","id":1735237,"avatar_url":"https://avatars.githubusercontent.com/u/1735237?v=3","gravatar_id":"","url":"https://api.github.com/users/dorlaor","html_url":"https://github.com/dorlaor","followers_url":"https://api.github.com/users/dorlaor/followers","following_url":"https://api.github.com/users/dorlaor/following{/other_user}","gists_url":"https://api.github.com/users/dorlaor/gists{/gist_id}","starred_url":"https://api.github.com/users/dorlaor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dorlaor/subscriptions","organizations_url":"https://api.github.com/users/dorlaor/orgs","repos_url":"https://api.github.com/users/dorlaor/repos","events_url":"https://api.github.com/users/dorlaor/events{/privacy}","received_events_url":"https://api.github.com/users/dorlaor/received_events","type":"User","site_admin":false},"open_issues":146,"closed_issues":54,"state":"open","created_at":"2015-06-15T06:37:52Z","updated_at":"2015-12-24T15:18:37Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2015-12-23T16:34:24Z","updated_at":"2015-12-25T10:09:28Z","closed_at":null,"body":"the iterative test does a sequence of add node and remove node and checks at the end that all the added keys are returned in select *\r\n\r\nhead\r\n\r\ncommit 8f4e90b87a1571533c3b15be4899ad5efe2d7170\r\nAuthor: Takuya ASADA <syuu@scylladb.com>\r\nDate:   Fri Dec 18 04:18:47 2015 +0900\r\n\r\n    dist: use tsc clocksource on AMI\r\n    \r\n    Stop using xen clocksource, use tsc clocksource instead.\r\n    Fixes #462\r\n    \r\n    Signed-off-by: Takuya ASADA <syuu@scylladb.com>\r\n\r\nIn some runs it fails\r\n```\r\nshlomi@localhost~/urchin-dtest (iterative_add_decomission)$ pkill -9 java ;rm -Rf ~/.dtest/*; export CASSANDRA_DIR=~/scylla; pkill -9 scylla; KEEP_TEST_DIR=true DEBUG=true PRINT_DEBUG=true nosetests -v -s update_cluster_layout_tests.py:TestUpdateClusterLayout.iterative_add_3_node_decommission_3_node_rf_1_test\r\niterative_add_3_node_decommission_3_node_rf_1_test (update_cluster_layout_tests.TestUpdateClusterLayout) ... cluster ccm directory: /home/shlomi/.dtest/dtest-WbSpPl\r\nadded node2\r\nadded node3\r\nadded node4\r\ndecommissioned node2\r\ndecommissioned node3\r\nadded node5\r\nadded node6\r\nadded node7\r\nFAIL\r\n\r\n======================================================================\r\nFAIL: iterative_add_3_node_decommission_3_node_rf_1_test (update_cluster_layout_tests.TestUpdateClusterLayout)\r\n----------------------------------------------------------------------\r\nTraceback (most recent call last):\r\n  File \"/home/shlomi/urchin-dtest/update_cluster_layout_tests.py\", line 165, in iterative_add_3_node_decommission_3_node_rf_1_test\r\n    self._iterative_add_decommission(node_count=3,iterations=2,rf=1)\r\n  File \"/home/shlomi/urchin-dtest/update_cluster_layout_tests.py\", line 134, in _iterative_add_decommission\r\n    self.assertEqual(len(result), iteration*node_count*100+1000, \"data loss after increasing size to %d expecting %d rows %d\" % (len(cluster.nodelist()),iteration*node_count*100+1000,len(result)))\r\nAssertionError: data loss after increasing size to 7 expecting 1600 rows 1568\r\n-------------------- >> begin captured logging << --------------------\r\ndtest: DEBUG: cluster ccm directory: /home/shlomi/.dtest/dtest-WbSpPl\r\ndtest: DEBUG: added node2\r\ndtest: DEBUG: added node3\r\ndtest: DEBUG: added node4\r\ndtest: DEBUG: decommissioned node2\r\ndtest: DEBUG: decommissioned node3\r\ndtest: DEBUG: added node5\r\ndtest: DEBUG: added node6\r\ndtest: DEBUG: added node7\r\n--------------------- >> end captured logging << ---------------------\r\n\r\n----------------------------------------------------------------------\r\nRan 1 test in 335.823s\r\n\r\nFAILED (failures=1)\r\n```"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/167218532","html_url":"https://github.com/scylladb/scylla/issues/701#issuecomment-167218532","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/701","id":167218532,"user":{"login":"asias","id":320076,"avatar_url":"https://avatars.githubusercontent.com/u/320076?v=3","gravatar_id":"","url":"https://api.github.com/users/asias","html_url":"https://github.com/asias","followers_url":"https://api.github.com/users/asias/followers","following_url":"https://api.github.com/users/asias/following{/other_user}","gists_url":"https://api.github.com/users/asias/gists{/gist_id}","starred_url":"https://api.github.com/users/asias/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/asias/subscriptions","organizations_url":"https://api.github.com/users/asias/orgs","repos_url":"https://api.github.com/users/asias/repos","events_url":"https://api.github.com/users/asias/events{/privacy}","received_events_url":"https://api.github.com/users/asias/received_events","type":"User","site_admin":false},"created_at":"2015-12-25T10:09:28Z","updated_at":"2015-12-25T10:09:28Z","body":"Running the test against c*. It passes without the sleep."}}