{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/rocksdb/issues/731","labels_url":"https://api.github.com/repos/facebook/rocksdb/issues/731/labels{/name}","comments_url":"https://api.github.com/repos/facebook/rocksdb/issues/731/comments","events_url":"https://api.github.com/repos/facebook/rocksdb/issues/731/events","html_url":"https://github.com/facebook/rocksdb/pull/731","id":107731259,"number":731,"title":"Allow storage backend to store metadata in manifest","user":{"login":"javigon","id":902977,"avatar_url":"https://avatars.githubusercontent.com/u/902977?v=3","gravatar_id":"","url":"https://api.github.com/users/javigon","html_url":"https://github.com/javigon","followers_url":"https://api.github.com/users/javigon/followers","following_url":"https://api.github.com/users/javigon/following{/other_user}","gists_url":"https://api.github.com/users/javigon/gists{/gist_id}","starred_url":"https://api.github.com/users/javigon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/javigon/subscriptions","organizations_url":"https://api.github.com/users/javigon/orgs","repos_url":"https://api.github.com/users/javigon/repos","events_url":"https://api.github.com/users/javigon/events{/privacy}","received_events_url":"https://api.github.com/users/javigon/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/facebook/rocksdb/labels/CLA%20Signed","name":"CLA Signed","color":"009900"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-09-22T14:52:56Z","updated_at":"2015-10-02T06:55:02Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/facebook/rocksdb/pulls/731","html_url":"https://github.com/facebook/rocksdb/pull/731","diff_url":"https://github.com/facebook/rocksdb/pull/731.diff","patch_url":"https://github.com/facebook/rocksdb/pull/731.patch"},"body":"Allow storage backend to store private metadata\r\n\r\nSummary: This patch provides a set of helper functions that would enable\r\na storage backend (e.g., posix, HDFS) to store sstable private metadata\r\nin the manifest. This frees the storage backend from implementing its\r\nown metadata mechanisms by reusing the functionality already present in\r\nthe manifest. This same idea can be applied to the WAL and MANIFEST; if\r\nthe mechanism is correct, patches for these will follow.\r\n\r\nTest Plan: This patch has been tested in posix and in a new backend for\r\nOpenChannel SSDs [1], where the private metadata functions have been\r\nimplemented. The idea is that testing functions are provided by the\r\nstorage backend when used.\r\n\r\n[1] https://github.com/OpenChannelSSD"},"comment":{"url":"https://api.github.com/repos/facebook/rocksdb/issues/comments/144938195","html_url":"https://github.com/facebook/rocksdb/pull/731#issuecomment-144938195","issue_url":"https://api.github.com/repos/facebook/rocksdb/issues/731","id":144938195,"user":{"login":"javigon","id":902977,"avatar_url":"https://avatars.githubusercontent.com/u/902977?v=3","gravatar_id":"","url":"https://api.github.com/users/javigon","html_url":"https://github.com/javigon","followers_url":"https://api.github.com/users/javigon/followers","following_url":"https://api.github.com/users/javigon/following{/other_user}","gists_url":"https://api.github.com/users/javigon/gists{/gist_id}","starred_url":"https://api.github.com/users/javigon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/javigon/subscriptions","organizations_url":"https://api.github.com/users/javigon/orgs","repos_url":"https://api.github.com/users/javigon/repos","events_url":"https://api.github.com/users/javigon/events{/privacy}","received_events_url":"https://api.github.com/users/javigon/received_events","type":"User","site_admin":false},"created_at":"2015-10-02T06:55:02Z","updated_at":"2015-10-02T06:55:02Z","body":"@dhruba Thanks for the proposal. The initial idea was to reuse the MANIFEST mechanisms to (i) avoid adding more complexity to the metadata scheme, and (ii) to take advantage of all the logic that ensure that sstables' and most logs' metadata make it to persistent storage. Seems to me that this approach would require maintaining a dedicated metadata I/O path, which probably would trigger more specific changes in the LSM logic. This said, I already added the logic to have two different directories (and storage backends) based on type file; this way I can store the superblock and other metadata that is not in the critical IO path (namely CURRENT, LOCK, and IDENTITY) in a normal ext partition using posix. I think this would be beneficial to other storage backends (e.g., HDFS); I will pack the changes in a patch and send them for you guys to look at them. In any case, if you guys think that a dedicated metadata path is the right way to do it, we can discuss about it.\r\n\r\n@siying Thanks for considering to prioritize this :) At the moment I add a new opcode (kPrivMeta) in the Tag enum; then the helper functions that you can see in this patch lead to a callback function in Env. For sstables I think that the approach is very clean since we already have struct FileMetadata. For the log it is a bit more intrusive (not in this patch) since (i) we do not have metadata more than the number, and (ii) the last log is not guaranteed to make it to the MANIFEST. It would be great to have a good way to do it, and maybe this new manifest format you are working on is the solution. Let me know if I can help with the manifest format, or if you want some input on it. I will prepare some other patches I intend to push upstream during these two weeks and put them in a branch for you to look at; maybe it helps having an overview of how to enable Env to save private metadata for different file types.\r\n"}}