{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/484","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/484/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/484/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/484/events","html_url":"https://github.com/scylladb/scylla/issues/484","id":112690717,"number":484,"title":"Add node update_cluster_layout_tests.py:TestUpdateClusterLayout.simple_add_node_2_test","user":{"login":"slivne","id":3465480,"avatar_url":"https://avatars.githubusercontent.com/u/3465480?v=3","gravatar_id":"","url":"https://api.github.com/users/slivne","html_url":"https://github.com/slivne","followers_url":"https://api.github.com/users/slivne/followers","following_url":"https://api.github.com/users/slivne/following{/other_user}","gists_url":"https://api.github.com/users/slivne/gists{/gist_id}","starred_url":"https://api.github.com/users/slivne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slivne/subscriptions","organizations_url":"https://api.github.com/users/slivne/orgs","repos_url":"https://api.github.com/users/slivne/repos","events_url":"https://api.github.com/users/slivne/events{/privacy}","received_events_url":"https://api.github.com/users/slivne/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420"}],"state":"open","locked":false,"assignee":{"login":"pdziepak","id":6995,"avatar_url":"https://avatars.githubusercontent.com/u/6995?v=3","gravatar_id":"","url":"https://api.github.com/users/pdziepak","html_url":"https://github.com/pdziepak","followers_url":"https://api.github.com/users/pdziepak/followers","following_url":"https://api.github.com/users/pdziepak/following{/other_user}","gists_url":"https://api.github.com/users/pdziepak/gists{/gist_id}","starred_url":"https://api.github.com/users/pdziepak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pdziepak/subscriptions","organizations_url":"https://api.github.com/users/pdziepak/orgs","repos_url":"https://api.github.com/users/pdziepak/repos","events_url":"https://api.github.com/users/pdziepak/events{/privacy}","received_events_url":"https://api.github.com/users/pdziepak/received_events","type":"User","site_admin":false},"milestone":null,"comments":17,"created_at":"2015-10-21T22:17:30Z","updated_at":"2015-12-15T09:24:21Z","closed_at":null,"body":"head\r\n\r\ncommit 5453cfbab75d7a9c968c0226020dfcda5277a028\r\nMerge: cf73413 3a95a9c\r\nAuthor: Avi Kivity <avi@scylladb.com>\r\nDate:   Wed Oct 21 08:59:42 2015 +0300\r\n\r\n    Merge \"snapshots: take + clear\" from Glauber\r\n    \r\n    \"This is the code for taking a snapshot, and clearing a snapshot.\"\r\n\r\nfails\r\n\r\nplease note you need to pull ccm, dtest to get this results\r\n```\r\nshlomi@localhost~/urchin-dtest (update_cluster_simple_add_node_tests)$ pkill -9 scylla;rm -Rf ~/.dtest/*; KEEP_TEST_DIR=true REUSE_CLUSTER=false DEBUG=true PRINT_DEBUG=true nosetests -v -s update_cluster_layout_tests.py:TestUpdateClusterLayout.simple_add_node_2_test\r\nsimple_add_node_2_test (update_cluster_layout_tests.TestUpdateClusterLayout) ... cluster ccm directory: /home/shlomi/.dtest/dtest-b1jr3Y\r\nFAIL\r\n\r\n======================================================================\r\nFAIL: simple_add_node_2_test (update_cluster_layout_tests.TestUpdateClusterLayout)\r\n----------------------------------------------------------------------\r\nTraceback (most recent call last):\r\n  File \"/home/shlomi/urchin-dtest/update_cluster_layout_tests.py\", line 159, in simple_add_node_2_test\r\n    self.assertEqual(node1_cache_entries + node2_cache_entries,1000, \"node1 cache %d node2 cache %d expected total of 1000\" % (node1_cache_entries,node2_cache_entries))\r\nAssertionError: node1 cache 1001 node2 cache 539 expected total of 1000\r\n-------------------- >> begin captured logging << --------------------\r\ndtest: DEBUG: cluster ccm directory: /home/shlomi/.dtest/dtest-b1jr3Y\r\n--------------------- >> end captured logging << ---------------------\r\n\r\n----------------------------------------------------------------------\r\nRan 1 test in 54.465s\r\n\r\nFAILED (failures=1)\r\n\r\n\r\n```\r\n\r\nThe test fails as it seems that when we stream ownership of some partitions we do not remove them from the row cache \r\n\r\nThis is problematic consider the following case:\r\n* node 1 has items A,B,C (all in cache)\r\n* node 2 started and gains ownership of item B,C (not deleting them from cache)\r\n* item B is deleted\r\n* node 2 runs a repair and compaction after gc_period - tombstone of B being deleted is lost\r\n* node 2 is removed passing item C\r\n* node 1 receives ownership of item C, yet in the cache it has a non existing item B and will return it on query\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/164697518","html_url":"https://github.com/scylladb/scylla/issues/484#issuecomment-164697518","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/484","id":164697518,"user":{"login":"slivne","id":3465480,"avatar_url":"https://avatars.githubusercontent.com/u/3465480?v=3","gravatar_id":"","url":"https://api.github.com/users/slivne","html_url":"https://github.com/slivne","followers_url":"https://api.github.com/users/slivne/followers","following_url":"https://api.github.com/users/slivne/following{/other_user}","gists_url":"https://api.github.com/users/slivne/gists{/gist_id}","starred_url":"https://api.github.com/users/slivne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slivne/subscriptions","organizations_url":"https://api.github.com/users/slivne/orgs","repos_url":"https://api.github.com/users/slivne/repos","events_url":"https://api.github.com/users/slivne/events{/privacy}","received_events_url":"https://api.github.com/users/slivne/received_events","type":"User","site_admin":false},"created_at":"2015-12-15T09:24:21Z","updated_at":"2015-12-15T09:24:21Z","body":"It may also do that because the streaming code in origin I think is in\nsstable level so it creates files on the disk but will not update the cache?\n\nIs out streaming code updating the cache ?, if it is - should it ?\n\nOn Tue, Dec 15, 2015 at 11:18 AM, Paweł Dziepak <notifications@github.com>\nwrote:\n\n> On 15 December 2015 at 10:13, Asias He <notifications@github.com> wrote:\n>\n> > On Tue, Dec 15, 2015 at 5:03 PM, Paweł Dziepak <notifications@github.com\n> >\n> > wrote:\n> >\n> > > Before this fix cache invalidation was in StreamReader.writeRow().\n> > >\n> >\n> > OK. When node1(source) streams data to node2(destination), the\n> > destination node will invalidate the row cache. The source node is not\n> > invalidated.\n> >\n> >\n> Yes, that should be sufficient. When bootstraping the node gains\n> responsibility over some ranges and ask their current owners to stream the\n> data. After that streaming is done the mapping of ranges to nodes is\n> updated in the cluster so that node that lost responsibility over a range\n> doesn't receive such requests anymore. When a node is decommissioned it may\n> regain responsibility over some ranges (and even after nodetool cleanup it\n> may still have some stale data in cache) but it will get these ranges via\n> streaming from the leaving node and after streaming is done invalidate its\n> cache thus removing any stale data.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/scylladb/scylla/issues/484#issuecomment-164696134>.\n>\n"}}