{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/676","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/676/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/676/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/676/events","html_url":"https://github.com/scylladb/scylla/issues/676","id":123229851,"number":676,"title":"compaction does not remove row tombstones ","user":{"login":"slivne","id":3465480,"avatar_url":"https://avatars.githubusercontent.com/u/3465480?v=3","gravatar_id":"","url":"https://api.github.com/users/slivne","html_url":"https://github.com/slivne","followers_url":"https://api.github.com/users/slivne/followers","following_url":"https://api.github.com/users/slivne/following{/other_user}","gists_url":"https://api.github.com/users/slivne/gists{/gist_id}","starred_url":"https://api.github.com/users/slivne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slivne/subscriptions","organizations_url":"https://api.github.com/users/slivne/orgs","repos_url":"https://api.github.com/users/slivne/repos","events_url":"https://api.github.com/users/slivne/events{/privacy}","received_events_url":"https://api.github.com/users/slivne/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420"}],"state":"open","locked":false,"assignee":{"login":"raphaelsc","id":1409139,"avatar_url":"https://avatars.githubusercontent.com/u/1409139?v=3","gravatar_id":"","url":"https://api.github.com/users/raphaelsc","html_url":"https://github.com/raphaelsc","followers_url":"https://api.github.com/users/raphaelsc/followers","following_url":"https://api.github.com/users/raphaelsc/following{/other_user}","gists_url":"https://api.github.com/users/raphaelsc/gists{/gist_id}","starred_url":"https://api.github.com/users/raphaelsc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raphaelsc/subscriptions","organizations_url":"https://api.github.com/users/raphaelsc/orgs","repos_url":"https://api.github.com/users/raphaelsc/repos","events_url":"https://api.github.com/users/raphaelsc/events{/privacy}","received_events_url":"https://api.github.com/users/raphaelsc/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/1","html_url":"https://github.com/scylladb/scylla/milestones/GA","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/1/labels","id":1164307,"number":1,"title":"GA","description":null,"creator":{"login":"dorlaor","id":1735237,"avatar_url":"https://avatars.githubusercontent.com/u/1735237?v=3","gravatar_id":"","url":"https://api.github.com/users/dorlaor","html_url":"https://github.com/dorlaor","followers_url":"https://api.github.com/users/dorlaor/followers","following_url":"https://api.github.com/users/dorlaor/following{/other_user}","gists_url":"https://api.github.com/users/dorlaor/gists{/gist_id}","starred_url":"https://api.github.com/users/dorlaor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dorlaor/subscriptions","organizations_url":"https://api.github.com/users/dorlaor/orgs","repos_url":"https://api.github.com/users/dorlaor/repos","events_url":"https://api.github.com/users/dorlaor/events{/privacy}","received_events_url":"https://api.github.com/users/dorlaor/received_events","type":"User","site_admin":false},"open_issues":163,"closed_issues":23,"state":"open","created_at":"2015-06-15T06:37:52Z","updated_at":"2015-12-21T08:21:06Z","due_on":null,"closed_at":null},"comments":8,"created_at":"2015-12-21T08:21:00Z","updated_at":"2015-12-21T11:36:52Z","closed_at":null,"body":"compaction does not remove delete row markers from sstable files\r\n\r\nto reproduce\r\n\r\n  1. ccm remove; ccm create scylla-1 --scylla --vnodes -n 1 --install-dir=/home/shlomi/urchin-ccm/../scylla; ccm start\r\n  2. ccm node1 -f c2\r\n```\r\nCREATE KEYSPACE keyspace1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '2'}  AND durable_writes = true;\r\n\r\nCREATE TABLE keyspace1.standard1 (\r\n    key blob PRIMARY KEY,\r\n    C0 blob,\r\n    C1 blob,\r\n    C2 blob,\r\n    C3 blob,\r\n    C4 blob\r\n) WITH bloom_filter_fp_chance = 0.01\r\n    AND caching = '{\"keys\":\"ALL\", \"rows_per_partition\":\"NONE\"}'\r\n    AND comment = ''\r\n    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}\r\n    AND compression = {}\r\n    AND dclocal_read_repair_chance = 0.1\r\n    AND default_time_to_live = 0\r\n    AND gc_grace_seconds = 0\r\n    AND max_index_interval = 2048\r\n    AND memtable_flush_period_in_ms = 0\r\n    AND min_index_interval = 128\r\n    AND read_repair_chance = 0.0\r\n    AND speculative_retry = '99.0PERCENTILE';\r\n\r\ninsert into keyspace1.standard1 (key,C0,C1) values (0x01,0x01,0x01);\r\ninsert into keyspace1.standard1 (key,C0,C1) values (0x02,0x01,0x01);\r\n```\r\n  3. ccm node1 flush\r\n  4. ccm node1 cqlsh -e \"delete from keyspace1.standard1 where key in (0x01);\"\r\n  5. ccm node1 flush\r\n  6. ccm node1 compact\r\n  7. cd into ~/.ccm/scylla-1/node1/data - and check content\r\n\r\nexport CASSANDRA_CONF=`pwd`/conf/ ; env cassandra.config=file://`pwd`/../conf/cassandra.yaml\r\n\r\n~/cassandra/tools/bin/sstable2json keyspace1/standard1-b57123a0a7b811e58f6f000000000000/keyspace1-standard1-ka-3-Data.db \r\n[\r\n{\"key\": \"02\",\r\n \"cells\": [[\"\",\"\",1450685393027190],\r\n           [\"c0\",\"01\",1450685393027190],\r\n           [\"c1\",\"01\",1450685393027190]]},\r\n{\"key\": \"01\",\r\n \"metadata\": {\"deletionInfo\": {\"markedForDeleteAt\":1450685406887001,\"localDeletionTime\":1450685406}},\r\n \"cells\": []}\r\n]\r\n\r\ndoing the same thing on cassandra leaves the data/keyspace/table data file with only the live data.\r\n\r\nThe test from origin compaction_test.py compaction_delete_test - does not cover this case.\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/166278620","html_url":"https://github.com/scylladb/scylla/issues/676#issuecomment-166278620","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/676","id":166278620,"user":{"login":"slivne","id":3465480,"avatar_url":"https://avatars.githubusercontent.com/u/3465480?v=3","gravatar_id":"","url":"https://api.github.com/users/slivne","html_url":"https://github.com/slivne","followers_url":"https://api.github.com/users/slivne/followers","following_url":"https://api.github.com/users/slivne/following{/other_user}","gists_url":"https://api.github.com/users/slivne/gists{/gist_id}","starred_url":"https://api.github.com/users/slivne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slivne/subscriptions","organizations_url":"https://api.github.com/users/slivne/orgs","repos_url":"https://api.github.com/users/slivne/repos","events_url":"https://api.github.com/users/slivne/events{/privacy}","received_events_url":"https://api.github.com/users/slivne/received_events","type":"User","site_admin":false},"created_at":"2015-12-21T11:36:52Z","updated_at":"2015-12-21T11:36:52Z","body":"Test \r\ncompaction_test.py:TestCompaction_with_LeveledCompactionStrategy.compaction_delete_2_test\r\ncompaction_test.py:TestCompaction_with_SizeTieredCompactionStrategy.compaction_delete_2_test"}}