{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/rocksdb/issues/642","labels_url":"https://api.github.com/repos/facebook/rocksdb/issues/642/labels{/name}","comments_url":"https://api.github.com/repos/facebook/rocksdb/issues/642/comments","events_url":"https://api.github.com/repos/facebook/rocksdb/issues/642/events","html_url":"https://github.com/facebook/rocksdb/issues/642","id":90488018,"number":642,"title":"Parallel Writes to mem tables, etc","user":{"login":"siying","id":5334567,"avatar_url":"https://avatars.githubusercontent.com/u/5334567?v=3","gravatar_id":"","url":"https://api.github.com/users/siying","html_url":"https://github.com/siying","followers_url":"https://api.github.com/users/siying/followers","following_url":"https://api.github.com/users/siying/following{/other_user}","gists_url":"https://api.github.com/users/siying/gists{/gist_id}","starred_url":"https://api.github.com/users/siying/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/siying/subscriptions","organizations_url":"https://api.github.com/users/siying/orgs","repos_url":"https://api.github.com/users/siying/repos","events_url":"https://api.github.com/users/siying/events{/privacy}","received_events_url":"https://api.github.com/users/siying/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"siying","id":5334567,"avatar_url":"https://avatars.githubusercontent.com/u/5334567?v=3","gravatar_id":"","url":"https://api.github.com/users/siying","html_url":"https://github.com/siying","followers_url":"https://api.github.com/users/siying/followers","following_url":"https://api.github.com/users/siying/following{/other_user}","gists_url":"https://api.github.com/users/siying/gists{/gist_id}","starred_url":"https://api.github.com/users/siying/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/siying/subscriptions","organizations_url":"https://api.github.com/users/siying/orgs","repos_url":"https://api.github.com/users/siying/repos","events_url":"https://api.github.com/users/siying/events{/privacy}","received_events_url":"https://api.github.com/users/siying/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2015-06-23T20:51:18Z","updated_at":"2015-06-25T17:57:47Z","closed_at":null,"body":"Let's use this issue to track the efforts to improve write throughput to mem tables.\r\n\r\nAs a discussion with @guyg8, I'll work on branch `sdong_write` and port necessary from branch `write_throughput`, which contains changes @guyg8 made. Hopefully branch `sdong_write`  can have a running codes after a week and let's see whether there can be any improvements."},"comment":{"url":"https://api.github.com/repos/facebook/rocksdb/issues/comments/115344744","html_url":"https://github.com/facebook/rocksdb/issues/642#issuecomment-115344744","issue_url":"https://api.github.com/repos/facebook/rocksdb/issues/642","id":115344744,"user":{"login":"siying","id":5334567,"avatar_url":"https://avatars.githubusercontent.com/u/5334567?v=3","gravatar_id":"","url":"https://api.github.com/users/siying","html_url":"https://github.com/siying","followers_url":"https://api.github.com/users/siying/followers","following_url":"https://api.github.com/users/siying/following{/other_user}","gists_url":"https://api.github.com/users/siying/gists{/gist_id}","starred_url":"https://api.github.com/users/siying/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/siying/subscriptions","organizations_url":"https://api.github.com/users/siying/orgs","repos_url":"https://api.github.com/users/siying/repos","events_url":"https://api.github.com/users/siying/events{/privacy}","received_events_url":"https://api.github.com/users/siying/received_events","type":"User","site_admin":false},"created_at":"2015-06-25T17:57:47Z","updated_at":"2015-06-25T17:57:47Z","body":"Let me report some status here.\r\n\r\nI built a prototype of parallel writer in memtable insert. The basic work flow is:\r\n\r\n1. The first writer (call leader in this context) wakes up and build the write batch group\r\n2. It writes updates of the write batch group to WAL\r\n3. It wakes up all writer threads for the write batch group\r\n4. Each thread woken independently update mem table for their updates. The concurrency of mem table and arena are solved using lock-free-skip-list and concurrent-arena built by @guyg8.\r\n\r\nSome initial results. \r\nI use value_size=0 to test effective of this approach. Having a value size bigger than 100 byte can easily generate more than 100MB/s ingestion rate even by single thread, so it's not the concern. I disabled automatic compaction so that all files will be piled as L0 files and the bottleneck is for sure mem table insert rate.\r\n\r\nHere is a sample benchmark query I run:\r\n\r\n./db_bench -benchmarks=fillrandom -threads=16 -batch_size=64 -memtablerep=\"lock_free_skip_list\" -value_size=0 --num=3000000 -level0_slowdown_writes_trigger=9999 -level0_stop_writes_trigger=9999 -disable_auto_compactions --max_write_buffer_number=8 -max_background_flushes=8\r\n\r\nHere is the observation. if -batch_size=1 (means we only update one key for one write request), multi-thread inserting to mem table is not helping at all. With write_batch=1, normal single writer skip list generates about 392K ops/s, or 6.0MB/s, while with 16 threads, it is even worse: 136K ops/s, 2.1 MB/s\r\n\r\nThe likely reason is the communication costs of those threads, like maintaining the waiting queue and wait for leader's instruction. Not sure how much we can optimize on codes' level.\r\n\r\nOn the other hand, if batch size is 8 or up, we can clearly see an improvement of total ingestion rate by using more threads (although not linearly). With -batch_size=64, 32 threads will generate 1.75M ops/s with 26.8MB/s, while single thread skip list can only generate 855K ops/s with 13.0MB. Worth noting that if running lock-free-skip-list in single writer mode, it can only generate 642K ops/s with 9.8MB.\r\n\r\nFinally, @guyg8 here is a question for you: I saw neither lock-free-skip-list, nor concurrent arena pass ThreadSanitizer checks. We usually keep our code base clean with ThreadSanitizer. I'm not sure whether they are false positive or real data race. But usually even if it is false positive, we rewrite the codes so that ThreadSanitizer can understand it is correct. Can you try to fix that? When you build RocksDB from clean you can set COMPILE_WITH_TSAN and it will be enabled."}}