{"action":"created","issue":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/4022","labels_url":"https://api.github.com/repos/influxdb/influxdb/issues/4022/labels{/name}","comments_url":"https://api.github.com/repos/influxdb/influxdb/issues/4022/comments","events_url":"https://api.github.com/repos/influxdb/influxdb/issues/4022/events","html_url":"https://github.com/influxdb/influxdb/issues/4022","id":105119543,"number":4022,"title":"[0.9.3] wal compacting causing database to time out","user":{"login":"kenzodeluxe","id":6376222,"avatar_url":"https://avatars.githubusercontent.com/u/6376222?v=3","gravatar_id":"","url":"https://api.github.com/users/kenzodeluxe","html_url":"https://github.com/kenzodeluxe","followers_url":"https://api.github.com/users/kenzodeluxe/followers","following_url":"https://api.github.com/users/kenzodeluxe/following{/other_user}","gists_url":"https://api.github.com/users/kenzodeluxe/gists{/gist_id}","starred_url":"https://api.github.com/users/kenzodeluxe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kenzodeluxe/subscriptions","organizations_url":"https://api.github.com/users/kenzodeluxe/orgs","repos_url":"https://api.github.com/users/kenzodeluxe/repos","events_url":"https://api.github.com/users/kenzodeluxe/events{/privacy}","received_events_url":"https://api.github.com/users/kenzodeluxe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"pauldix","id":4331,"avatar_url":"https://avatars.githubusercontent.com/u/4331?v=3","gravatar_id":"","url":"https://api.github.com/users/pauldix","html_url":"https://github.com/pauldix","followers_url":"https://api.github.com/users/pauldix/followers","following_url":"https://api.github.com/users/pauldix/following{/other_user}","gists_url":"https://api.github.com/users/pauldix/gists{/gist_id}","starred_url":"https://api.github.com/users/pauldix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pauldix/subscriptions","organizations_url":"https://api.github.com/users/pauldix/orgs","repos_url":"https://api.github.com/users/pauldix/repos","events_url":"https://api.github.com/users/pauldix/events{/privacy}","received_events_url":"https://api.github.com/users/pauldix/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/influxdb/influxdb/milestones/43","html_url":"https://github.com/influxdb/influxdb/milestones/0.9.4","labels_url":"https://api.github.com/repos/influxdb/influxdb/milestones/43/labels","id":1172852,"number":43,"title":"0.9.4","description":"","creator":{"login":"otoolep","id":536312,"avatar_url":"https://avatars.githubusercontent.com/u/536312?v=3","gravatar_id":"","url":"https://api.github.com/users/otoolep","html_url":"https://github.com/otoolep","followers_url":"https://api.github.com/users/otoolep/followers","following_url":"https://api.github.com/users/otoolep/following{/other_user}","gists_url":"https://api.github.com/users/otoolep/gists{/gist_id}","starred_url":"https://api.github.com/users/otoolep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/otoolep/subscriptions","organizations_url":"https://api.github.com/users/otoolep/orgs","repos_url":"https://api.github.com/users/otoolep/repos","events_url":"https://api.github.com/users/otoolep/events{/privacy}","received_events_url":"https://api.github.com/users/otoolep/received_events","type":"User","site_admin":false},"open_issues":66,"closed_issues":75,"state":"open","created_at":"2015-06-19T18:25:47Z","updated_at":"2015-09-10T18:41:27Z","due_on":"2015-09-10T07:00:00Z","closed_at":null},"comments":5,"created_at":"2015-09-06T22:05:57Z","updated_at":"2015-09-10T20:07:13Z","closed_at":null,"body":"Once a database has reached a critical size (11GB was mentioned in another issue and seems relevant somehow), wal flushing is causing outages:\r\n```\r\n[http] 2015/09/06 23:56:55 1.2.3.4 - - [06/Sep/2015:23:56:55 +0200] POST /write?db=monitoring&precision=s HTTP/1.1 204 0 - client 20.828986ms\r\n[wal] 2015/09/06 23:56:56 Flush due to threshold. Flushing 982 series with 5260667 bytes from partition 1. Compacting 702 series\r\n(..)\r\n[http] 2015/09/06 23:57:01 1.2.3.4 - - [06/Sep/2015:23:56:56 +0200] POST /write?db=monitoring&precision=s HTTP/1.1 500 32 - client 5.007814203s\r\n(...)\r\n[http] 2015/09/07 00:02:34 1.2.3.4 - - [07/Sep/2015:00:02:29 +0200] POST /write?db=monitoring&precision=s HTTP/1.1 500 32 - client 5.008541152s\r\n\r\n```\r\nThe database will never recover from this fault until it's removed or wal is deleted. Will then happen again once wal is flushed again.\r\ndu -hs:\r\n55G\t/var/opt/influxdb/data/monitoring"},"comment":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/comments/139365546","html_url":"https://github.com/influxdb/influxdb/issues/4022#issuecomment-139365546","issue_url":"https://api.github.com/repos/influxdb/influxdb/issues/4022","id":139365546,"user":{"login":"otoolep","id":536312,"avatar_url":"https://avatars.githubusercontent.com/u/536312?v=3","gravatar_id":"","url":"https://api.github.com/users/otoolep","html_url":"https://github.com/otoolep","followers_url":"https://api.github.com/users/otoolep/followers","following_url":"https://api.github.com/users/otoolep/following{/other_user}","gists_url":"https://api.github.com/users/otoolep/gists{/gist_id}","starred_url":"https://api.github.com/users/otoolep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/otoolep/subscriptions","organizations_url":"https://api.github.com/users/otoolep/orgs","repos_url":"https://api.github.com/users/otoolep/repos","events_url":"https://api.github.com/users/otoolep/events{/privacy}","received_events_url":"https://api.github.com/users/otoolep/received_events","type":"User","site_admin":false},"created_at":"2015-09-10T20:07:13Z","updated_at":"2015-09-10T20:07:13Z","body":"0.9.4-rc1 contains the change that writes will start failing if memory usage gets too high. \r\n\r\n  https://s3.amazonaws.com/influxdb/influxdb_0.9.4-rc1_amd64.deb\r\n  https://s3.amazonaws.com/influxdb/influxdb-0.9.4_rc1-1.x86_64.rpm\r\n"}}