{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/15481","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/15481/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/15481/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/15481/events","html_url":"https://github.com/elastic/elasticsearch/issues/15481","id":122525016,"number":15481,"title":"Segment merges should try to avoid exhausting all free disk space","user":{"login":"mikemccand","id":796508,"avatar_url":"https://avatars.githubusercontent.com/u/796508?v=3","gravatar_id":"","url":"https://api.github.com/users/mikemccand","html_url":"https://github.com/mikemccand","followers_url":"https://api.github.com/users/mikemccand/followers","following_url":"https://api.github.com/users/mikemccand/following{/other_user}","gists_url":"https://api.github.com/users/mikemccand/gists{/gist_id}","starred_url":"https://api.github.com/users/mikemccand/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikemccand/subscriptions","organizations_url":"https://api.github.com/users/mikemccand/orgs","repos_url":"https://api.github.com/users/mikemccand/repos","events_url":"https://api.github.com/users/mikemccand/events{/privacy}","received_events_url":"https://api.github.com/users/mikemccand/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Core","name":":Core","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/bug","name":"bug","color":"9e2c2c"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/PITA","name":"PITA","color":"e11d21"}],"state":"open","locked":false,"assignee":{"login":"mikemccand","id":796508,"avatar_url":"https://avatars.githubusercontent.com/u/796508?v=3","gravatar_id":"","url":"https://api.github.com/users/mikemccand","html_url":"https://github.com/mikemccand","followers_url":"https://api.github.com/users/mikemccand/followers","following_url":"https://api.github.com/users/mikemccand/following{/other_user}","gists_url":"https://api.github.com/users/mikemccand/gists{/gist_id}","starred_url":"https://api.github.com/users/mikemccand/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikemccand/subscriptions","organizations_url":"https://api.github.com/users/mikemccand/orgs","repos_url":"https://api.github.com/users/mikemccand/repos","events_url":"https://api.github.com/users/mikemccand/events{/privacy}","received_events_url":"https://api.github.com/users/mikemccand/received_events","type":"User","site_admin":false},"milestone":null,"comments":4,"created_at":"2015-12-16T15:13:41Z","updated_at":"2015-12-21T20:04:47Z","closed_at":null,"body":"When a Lucene segment merge runs, it needs sizable free temporary disk space to do its work.  If it runs out of disk space part way through, the OS rejects the `write` call, and we'll hit a \"tragic\" exception that will fail that shard.\r\n\r\nThis behavior is particularly evil because on filling up the disk, the merge will then go and remove all temp files, thus freeing up (sizable) disk space and provoking issues like #15420 where \"at first it failed and now it succeeded\" leads to corruption.\r\n\r\nInstead, we should avoid kicking off the merge, up front, if there isn't enough space?  I think we need some simple \"reservation\" API where the merge can claim how much space it expects to use, and fail at that point if there isn't enough space.\r\n\r\nProbably we could do this with a `Directory` wrapper that 1) uses Java's APIs to monitor free space periodically, and 2) adds this new reservation API.  This wrapper should then throw the \"disk full\" `IOException` long before the underlying disk actually fills up.  We would need to wrap `IndexOutput` as well, because the up front estimate could be off, and of course other things (merges from other shards, users copying files around, etc.) can concurrently fill up the disk.\r\n\r\nI think much of this improvement could be done in Lucene, with ES customizing the \"reserve\" method to be aware of N shards that are all reserving space for their merges on one node?  Lucene's default `TieredMergePolicy` already estimates the size of the merged segment to enforce its `maxMergedSegmentMB`...\r\n\r\nOr maybe there's a simpler approach?"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/166404767","html_url":"https://github.com/elastic/elasticsearch/issues/15481#issuecomment-166404767","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/15481","id":166404767,"user":{"login":"rmuir","id":504194,"avatar_url":"https://avatars.githubusercontent.com/u/504194?v=3","gravatar_id":"","url":"https://api.github.com/users/rmuir","html_url":"https://github.com/rmuir","followers_url":"https://api.github.com/users/rmuir/followers","following_url":"https://api.github.com/users/rmuir/following{/other_user}","gists_url":"https://api.github.com/users/rmuir/gists{/gist_id}","starred_url":"https://api.github.com/users/rmuir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rmuir/subscriptions","organizations_url":"https://api.github.com/users/rmuir/orgs","repos_url":"https://api.github.com/users/rmuir/repos","events_url":"https://api.github.com/users/rmuir/events{/privacy}","received_events_url":"https://api.github.com/users/rmuir/received_events","type":"User","site_admin":false},"created_at":"2015-12-21T20:04:47Z","updated_at":"2015-12-21T20:04:47Z","body":"I am -1 to stopping merging because space runs out. Failing the merge is the right thing to do.\r\n\r\nI do not think we should add \"reserve space\" in lucene unless we back it by a method in the JDK that actually reserves space (http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_fallocate.html or similar), and lucene does not need this since we only rely on atomic rename always. The problem is not lucene corrupting anything here.\r\n\r\nAs far as testing, I still have a disk full branch if anyone wants to help: https://github.com/apache/lucene-solr/compare/trunk...rmuir:diskfull\r\n\r\nTesting this well is easy with something like lucene that uses a very contained set of the java filesystem api and is only write-once, it needs lots of improvements to implement the other apis. For example, lucene does not even use FileChannel for write, just for locking. But this is more complex than an outputstream for testing this.\r\n"}}