{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/rocksdb/issues/616","labels_url":"https://api.github.com/repos/facebook/rocksdb/issues/616/labels{/name}","comments_url":"https://api.github.com/repos/facebook/rocksdb/issues/616/comments","events_url":"https://api.github.com/repos/facebook/rocksdb/issues/616/events","html_url":"https://github.com/facebook/rocksdb/issues/616","id":81393785,"number":616,"title":"Iterator on WriteBatchWithIndex sometimes does not return all keys","user":{"login":"adamretter","id":1264057,"avatar_url":"https://avatars.githubusercontent.com/u/1264057?v=3","gravatar_id":"","url":"https://api.github.com/users/adamretter","html_url":"https://github.com/adamretter","followers_url":"https://api.github.com/users/adamretter/followers","following_url":"https://api.github.com/users/adamretter/following{/other_user}","gists_url":"https://api.github.com/users/adamretter/gists{/gist_id}","starred_url":"https://api.github.com/users/adamretter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adamretter/subscriptions","organizations_url":"https://api.github.com/users/adamretter/orgs","repos_url":"https://api.github.com/users/adamretter/repos","events_url":"https://api.github.com/users/adamretter/events{/privacy}","received_events_url":"https://api.github.com/users/adamretter/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":28,"created_at":"2015-05-27T10:40:45Z","updated_at":"2015-09-02T00:34:44Z","closed_at":null,"body":"I am seeing a strange problem which I am struggling to track down. Unfortunately the problem is non-deterministic, I have a suite of tests and if I run them 5 times, then the problem may only occur 3 times; It seems that ***sometimes*** a NewIteratorWithBase taken on a Column Family for a WriteBatchWithIndex can miss some keys. \r\n\r\nFor background, I am working from Java and I have a WriteBatchWithIndex for which I take a NewIteratorWithBase on a Column Family. My column family holds serialized objects (of key/value pairs), the keys I store are composite with three parts\r\n\r\n1. object id (a UUID 128bits)\r\n2. unique key unit decoding length (stored as 4 bytes)\r\n3. unique key id (a hierarchical id like 1.1 or 1.1.1) which is decoded by using 2. so that a bytewise comparison will work.\r\n\r\nI have a custom comparator written in C++ which provides the ordering for my keys. It groups keys together by object id, by using memcmp on two different object ids, it also knows how to order the unique keys so that for example 1.1 comes before 1.1.1, and 1.1.2 comes before 1.2. This comparator is provided to the column family descriptor when opening the  database, the returned column family handle is used when I take a NewIteratorWithBase.\r\n\r\nWhen tracing my application, when I see the expected behavior the following events occur:\r\n\r\n```\r\nStoring object: 749b500f-9701-411e-a2a2-112bb9c855be\r\nStored: 749b500f-9701-411e-a2a2-112bb9c855be:1\r\nStored: 749b500f-9701-411e-a2a2-112bb9c855be:1.1\r\nStored: 749b500f-9701-411e-a2a2-112bb9c855be:1.2\r\nStored: 749b500f-9701-411e-a2a2-112bb9c855be:1.2.1\r\n\r\n....\r\nRemoving object: 749b500f-9701-411e-a2a2-112bb9c855be\r\nRemoved: 749b500f-9701-411e-a2a2-112bb9c855be:1\r\nRemoved: 749b500f-9701-411e-a2a2-112bb9c855be:1.1\r\nRemoved: 749b500f-9701-411e-a2a2-112bb9c855be:1.2\r\nRemoved: 749b500f-9701-411e-a2a2-112bb9c855be:1.2.1\r\n```\r\n\r\nHowever, when I get errors, the trace shows that the following events occured:\r\n\r\n```\r\nStoring object: 749b500f-9701-411e-a2a2-112bb9c855be\r\nStored: 749b500f-9701-411e-a2a2-112bb9c855be:1\r\nStored: 749b500f-9701-411e-a2a2-112bb9c855be:1.1\r\nStored: 749b500f-9701-411e-a2a2-112bb9c855be:1.2\r\nStored: 749b500f-9701-411e-a2a2-112bb9c855be:1.2.1\r\n\r\n....\r\nRemoving object: 749b500f-9701-411e-a2a2-112bb9c855be\r\nRemoved: 749b500f-9701-411e-a2a2-112bb9c855be:1\r\nRemoved: 749b500f-9701-411e-a2a2-112bb9c855be:1.1\r\nRemoved: 749b500f-9701-411e-a2a2-112bb9c855be:1.2\r\n```\r\n***Note*** that the key/value pair `749b500f-9701-411e-a2a2-112bb9c855be:1.2.1` is not removed in the second trace.\r\n\r\nMy code for removing an object looks like:\r\n\r\n```java\r\nObjectId objectId = object.getId();\r\nrocksLog.trace(\"Removing object: \" + objectId.toString());\r\n\r\nRocksIterator baseIterator = rocks.newIterator(cf, readOptions);\r\nRocksIterator iterator = writeBatchWithIndex.newIteratorWithBase(cf, baseIterator);\r\n\r\n//seek to our objectId, iterate over all following key/value pairs that start with our objectId\r\nfor (iterator.seek(objectId.serialize()); iterator.isValid() && ObjectKey.deserializeObjectId(iterator.key(), 0).startsWith(objectId); iterator.next()) {\r\n    writeBatchWithIndex.remove(cf, iterator.key());\r\n    rocksLog.trace(\"Removed: \" + ObjectKey.deserialize(iterator.key(), 0).toString());\r\n}\r\n```\r\n\r\nAny ideas why I might see NewIteratorWithBase not always iterate all present keys? The thing that worries me is that running the same sequence of store and remove steps over and over again only shows the problem sometimes and not every time. Could this be a problem in my comparator, or is it more likely the problem is elsewhere in Rocks? I can share the comparator code if that helps.\r\n\r\nPerhaps this is a known issue that is already fixed? I am using revision `4a855c0` of RocksDB from May 18th 2015.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/facebook/rocksdb/issues/comments/136902753","html_url":"https://github.com/facebook/rocksdb/issues/616#issuecomment-136902753","issue_url":"https://api.github.com/repos/facebook/rocksdb/issues/616","id":136902753,"user":{"login":"agiardullo","id":10343516,"avatar_url":"https://avatars.githubusercontent.com/u/10343516?v=3","gravatar_id":"","url":"https://api.github.com/users/agiardullo","html_url":"https://github.com/agiardullo","followers_url":"https://api.github.com/users/agiardullo/followers","following_url":"https://api.github.com/users/agiardullo/following{/other_user}","gists_url":"https://api.github.com/users/agiardullo/gists{/gist_id}","starred_url":"https://api.github.com/users/agiardullo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/agiardullo/subscriptions","organizations_url":"https://api.github.com/users/agiardullo/orgs","repos_url":"https://api.github.com/users/agiardullo/repos","events_url":"https://api.github.com/users/agiardullo/events{/privacy}","received_events_url":"https://api.github.com/users/agiardullo/received_events","type":"User","site_admin":false},"created_at":"2015-09-02T00:34:44Z","updated_at":"2015-09-02T00:34:44Z","body":"Can this issue be closed now? "}}