{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/rocksdb/issues/719","labels_url":"https://api.github.com/repos/facebook/rocksdb/issues/719/labels{/name}","comments_url":"https://api.github.com/repos/facebook/rocksdb/issues/719/comments","events_url":"https://api.github.com/repos/facebook/rocksdb/issues/719/events","html_url":"https://github.com/facebook/rocksdb/issues/719","id":106443105,"number":719,"title":"Sudden memory growth during RocksDB compaction ","user":{"login":"JeevithaK","id":14283646,"avatar_url":"https://avatars.githubusercontent.com/u/14283646?v=3","gravatar_id":"","url":"https://api.github.com/users/JeevithaK","html_url":"https://github.com/JeevithaK","followers_url":"https://api.github.com/users/JeevithaK/followers","following_url":"https://api.github.com/users/JeevithaK/following{/other_user}","gists_url":"https://api.github.com/users/JeevithaK/gists{/gist_id}","starred_url":"https://api.github.com/users/JeevithaK/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JeevithaK/subscriptions","organizations_url":"https://api.github.com/users/JeevithaK/orgs","repos_url":"https://api.github.com/users/JeevithaK/repos","events_url":"https://api.github.com/users/JeevithaK/events{/privacy}","received_events_url":"https://api.github.com/users/JeevithaK/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-09-14T22:46:04Z","updated_at":"2015-09-16T00:06:03Z","closed_at":null,"body":"Hi,\r\n\r\nI have a rocksdb with ~280GB data and our process is running with memory limit set to 16G. Every few days, our process is getting killed by oom-killer. \r\nAccording to the heap profiles collected just before crash: ~75% of memory is used by TableIndexReader & Filter data created as a result of BackgroundCompaction.\r\n\r\nIncreasing the memory limit to 24G didn't help either. It eventually hits the limit and crashes.\r\nIs there a way to limit the memory usage by rocksdb compaction?\r\n\r\nHere is the list of DB options we use:\r\ntable_options.filter_policy.reset(NewBloomFilterPolicy(8, false /* use_block_based_builder */));\r\n\r\ntable_options.block_size = 8 * 1024;\r\ntable_options.block_cache = NewLRUCache(128 * 1024 * 1024);\r\n\r\noptions->min_write_buffer_number_to_merge = 3;\r\noptions->max_write_buffer_number = 5;\r\n\r\noptions->env->SetBackgroundThreads(3, Env::Priority::HIGH);\r\noptions->max_background_flushes = 3;\r\n\r\noptions->env->SetBackgroundThreads(4, Env::Priority::LOW);\r\noptions->max_background_compactions = 4;\r\n\r\noptions->level0_file_num_compaction_trigger = 8;\r\noptions->level0_slowdown_writes_trigger = 17;\r\noptions->level0_stop_writes_trigger = 24;\r\noptions->target_file_size_base = 64 * 1024 * 1024;\r\noptions->max_bytes_for_level_base = 8 * options->target_file_size_base;\r\noptions->max_bytes_for_level_multiplier = 8;\r\noptions->target_file_size_multiplier = 3;\r\n\r\nThanks!"},"comment":{"url":"https://api.github.com/repos/facebook/rocksdb/issues/comments/140584572","html_url":"https://github.com/facebook/rocksdb/issues/719#issuecomment-140584572","issue_url":"https://api.github.com/repos/facebook/rocksdb/issues/719","id":140584572,"user":{"login":"JeevithaK","id":14283646,"avatar_url":"https://avatars.githubusercontent.com/u/14283646?v=3","gravatar_id":"","url":"https://api.github.com/users/JeevithaK","html_url":"https://github.com/JeevithaK","followers_url":"https://api.github.com/users/JeevithaK/followers","following_url":"https://api.github.com/users/JeevithaK/following{/other_user}","gists_url":"https://api.github.com/users/JeevithaK/gists{/gist_id}","starred_url":"https://api.github.com/users/JeevithaK/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JeevithaK/subscriptions","organizations_url":"https://api.github.com/users/JeevithaK/orgs","repos_url":"https://api.github.com/users/JeevithaK/repos","events_url":"https://api.github.com/users/JeevithaK/events{/privacy}","received_events_url":"https://api.github.com/users/JeevithaK/received_events","type":"User","site_admin":false},"created_at":"2015-09-16T00:06:03Z","updated_at":"2015-09-16T00:06:03Z","body":"Hi Igor,\r\n\r\nReally appreciate your quick response!\r\n\r\nFew months ago, we had similar memory growth problems and we tried changing the block size to 32K. Since that resulted in performance degradation we ended up increasing the process memory limit and went back to 8K. I will change the block size to 16K and check if the performance is acceptable.\r\n\r\nSome questions for our understanding: \r\n1. Since we are leaving cache_index_and_filter_blocks=false, is it fair to assume that the index and filter data brought into memory stays in memory as long as the corresponding SSTable is open?\r\n2. In steady state, (i.e., when there are no compactions and light load of reads & writes), the memory usage of our process stays around 8 - 9GB. It is only when compaction starts (especially at L4), we see the memory spiking up. Our current hypotheses is: it is because the indexes and filters of new SSTs being created as part of the compaction and the old SSTs are both in memory. Does it sound correct?\r\n3. Do you have any suggestions on estimating the peak memory usage which accounts for any extra memory consumption by compaction?\r\n\r\nThank you!"}}