{"action":"created","issue":{"url":"https://api.github.com/repos/bitly/nsq/issues/625","labels_url":"https://api.github.com/repos/bitly/nsq/issues/625/labels{/name}","comments_url":"https://api.github.com/repos/bitly/nsq/issues/625/comments","events_url":"https://api.github.com/repos/bitly/nsq/issues/625/events","html_url":"https://github.com/bitly/nsq/pull/625","id":99917492,"number":625,"title":"[dev] nsqd: per-topic WAL","user":{"login":"mreiferson","id":187441,"avatar_url":"https://avatars.githubusercontent.com/u/187441?v=3","gravatar_id":"","url":"https://api.github.com/users/mreiferson","html_url":"https://github.com/mreiferson","followers_url":"https://api.github.com/users/mreiferson/followers","following_url":"https://api.github.com/users/mreiferson/following{/other_user}","gists_url":"https://api.github.com/users/mreiferson/gists{/gist_id}","starred_url":"https://api.github.com/users/mreiferson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mreiferson/subscriptions","organizations_url":"https://api.github.com/users/mreiferson/orgs","repos_url":"https://api.github.com/users/mreiferson/repos","events_url":"https://api.github.com/users/mreiferson/events{/privacy}","received_events_url":"https://api.github.com/users/mreiferson/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/bitly/nsq/labels/feature","name":"feature","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-08-09T18:11:09Z","updated_at":"2015-08-10T15:31:23Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/bitly/nsq/pulls/625","html_url":"https://github.com/bitly/nsq/pull/625","diff_url":"https://github.com/bitly/nsq/pull/625.diff","patch_url":"https://github.com/bitly/nsq/pull/625.patch"},"body":"This introduces a write-ahead log for `nsqd` topics using https://github.com/mreiferson/wal.  This is the first part of #510.\r\n\r\nAt a high level, the WAL is a filesystem backed FIFO queue with monotonically increasing IDs, optional periodic flushing semantics (off by default), and the ability to open a \"cursor\" at any index within its retention window.  Many cursors can be open simultaneously at different indices.\r\n\r\nEach topic has its own WAL and each channel has its own cursor into it.  The filesystem replaces the per-topic goroutine `messagePump()` that previously copied messages to channels.  Instead, we use Go's low-level [`sync.Condition`](http://golang.org/pkg/sync/#Cond) to \"wake up\" any open cursors so they can advance when events are appended.\r\n\r\nThere are quite a few remaining items to figure out here, namely:\r\n\r\n- [ ] proper depth accounting\r\n   - [ ] persistence of state to survive restarts\r\n- [ ] what happens to `BackendDepth`?\r\n- [ ] `DPUB`\r\n- [ ] topic pausing\r\n- [ ] ephemeral topics/channels\r\n- [ ] msg `attempts` accounting\r\n- [ ] proper retention threshold for persisted messages"},"comment":{"url":"https://api.github.com/repos/bitly/nsq/issues/comments/129497372","html_url":"https://github.com/bitly/nsq/pull/625#issuecomment-129497372","issue_url":"https://api.github.com/repos/bitly/nsq/issues/625","id":129497372,"user":{"login":"mreiferson","id":187441,"avatar_url":"https://avatars.githubusercontent.com/u/187441?v=3","gravatar_id":"","url":"https://api.github.com/users/mreiferson","html_url":"https://github.com/mreiferson","followers_url":"https://api.github.com/users/mreiferson/followers","following_url":"https://api.github.com/users/mreiferson/following{/other_user}","gists_url":"https://api.github.com/users/mreiferson/gists{/gist_id}","starred_url":"https://api.github.com/users/mreiferson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mreiferson/subscriptions","organizations_url":"https://api.github.com/users/mreiferson/orgs","repos_url":"https://api.github.com/users/mreiferson/repos","events_url":"https://api.github.com/users/mreiferson/events{/privacy}","received_events_url":"https://api.github.com/users/mreiferson/received_events","type":"User","site_admin":false},"created_at":"2015-08-10T15:31:23Z","updated_at":"2015-08-10T15:31:23Z","body":"NOTE: there is still back-pressure from the periodic syncing because `DiskQueue` cannot make progress during a sync and individual writes are synchronized with the `PUB`."}}