{"action":"opened","issue":{"url":"https://api.github.com/repos/facebook/rocksdb/issues/493","labels_url":"https://api.github.com/repos/facebook/rocksdb/issues/493/labels{/name}","comments_url":"https://api.github.com/repos/facebook/rocksdb/issues/493/comments","events_url":"https://api.github.com/repos/facebook/rocksdb/issues/493/events","html_url":"https://github.com/facebook/rocksdb/issues/493","id":56358964,"number":493,"title":"Improving compaction performance","user":{"login":"ekg","id":145425,"avatar_url":"https://avatars.githubusercontent.com/u/145425?v=3","gravatar_id":"","url":"https://api.github.com/users/ekg","html_url":"https://github.com/ekg","followers_url":"https://api.github.com/users/ekg/followers","following_url":"https://api.github.com/users/ekg/following{/other_user}","gists_url":"https://api.github.com/users/ekg/gists{/gist_id}","starred_url":"https://api.github.com/users/ekg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ekg/subscriptions","organizations_url":"https://api.github.com/users/ekg/orgs","repos_url":"https://api.github.com/users/ekg/repos","events_url":"https://api.github.com/users/ekg/events{/privacy}","received_events_url":"https://api.github.com/users/ekg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-02-03T10:24:06Z","updated_at":"2015-02-03T10:24:06Z","closed_at":null,"body":"I'm working with data of around a terabyte. I can now efficiently load this into rocksdb using configuration as described in #490. The happy result is that the process is now bottlenecked at compaction.\r\n\r\nThe data load can run at nearly 100M/s, but I notice that compaction runs at only 10M/s. I have a lot of free cores, and at least 200M/s of available write, so I was wondering if there is a way to improve throughput.\r\n\r\nIn principle the db could be compacted in chunks in parallel, to reduce the number of input SSTs to the final merge-sort at the cost of a lot more compute and IO (I wouldn't mind either as I'm stuck waiting on iterations of this process). However, it's not clear that this gets at the underlying problem.\r\n\r\nHow can I improve compaction performance? Are there any settings in options which could be used, or is this a deeper limitation? What are the dominant factors in compaction runtime?"}}