{"action":"created","issue":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/4516","labels_url":"https://api.github.com/repos/influxdb/influxdb/issues/4516/labels{/name}","comments_url":"https://api.github.com/repos/influxdb/influxdb/issues/4516/comments","events_url":"https://api.github.com/repos/influxdb/influxdb/issues/4516/events","html_url":"https://github.com/influxdb/influxdb/pull/4516","id":112468697,"number":4516,"title":"Refactor hinted-handoff service","user":{"login":"otoolep","id":536312,"avatar_url":"https://avatars.githubusercontent.com/u/536312?v=3","gravatar_id":"","url":"https://api.github.com/users/otoolep","html_url":"https://github.com/otoolep","followers_url":"https://api.github.com/users/otoolep/followers","following_url":"https://api.github.com/users/otoolep/following{/other_user}","gists_url":"https://api.github.com/users/otoolep/gists{/gist_id}","starred_url":"https://api.github.com/users/otoolep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/otoolep/subscriptions","organizations_url":"https://api.github.com/users/otoolep/orgs","repos_url":"https://api.github.com/users/otoolep/repos","events_url":"https://api.github.com/users/otoolep/events{/privacy}","received_events_url":"https://api.github.com/users/otoolep/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/influxdb/influxdb/labels/2%20-%20Working","name":"2 - Working","color":"CCCCCC"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-10-20T21:55:49Z","updated_at":"2015-10-27T21:42:52Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/influxdb/influxdb/pulls/4516","html_url":"https://github.com/influxdb/influxdb/pull/4516","diff_url":"https://github.com/influxdb/influxdb/pull/4516.diff","patch_url":"https://github.com/influxdb/influxdb/pull/4516.patch"},"body":"This change refactors the hinted-handoff service.\r\n\r\nWhereas previously the system repeatedly launched a short-lived goroutine per active node (to drain all HH data for that node), it now launches a single long-running goroutine per active node which remains running until the node processor is purged due to being marked inactive (which may never happen). A node is considered active until it is DROPped.\r\n\r\nGiving each target node a dedicated, long-running, goroutine makes it easier to deal with some HH edge cases. \r\n\r\nThe change also cleans up stats, and adds HH diagnostics.\r\n\r\n```\r\n> show diagnostics for 'hh'\r\nname: hh\r\n--------\r\nnode    active  last modified                           head                            tail\r\n2       yes     2015-10-26T16:33:48.579769662-07:00     /tmp/influxdb8086/hh/2/1:0      /tmp/influxdb8086/hh/2/1:43\r\n3       yes     2015-10-26T16:34:02.055769467-07:00     /tmp/influxdb8086/hh/3/1:0      /tmp/influxdb8086/hh/3/1:2029\r\n\r\n\r\n> show stats\r\nname: hh\r\ntags: path=/tmp/influxdb8086/hh\r\nwr_shard_req    wr_shard_req_points\r\n------------    -------------------\r\n154             154\r\n\r\nname: hh_processor\r\ntags: node=3, path=/tmp/influxdb8086/hh/3\r\nwr_node_req_fail        wr_shard_req    wr_shard_req_points\r\n----------------        ------------    -------------------\r\n1                       154             154\r\n```\r\n\r\nMultiple rounds of testing done with 3 nodes, with the 2 nodes not receiving the write requests killed every 10 seconds, multiple times. After these 2 nodes were finally brought online and kept online, they received all the missed writes."},"comment":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/comments/151654323","html_url":"https://github.com/influxdb/influxdb/pull/4516#issuecomment-151654323","issue_url":"https://api.github.com/repos/influxdb/influxdb/issues/4516","id":151654323,"user":{"login":"otoolep","id":536312,"avatar_url":"https://avatars.githubusercontent.com/u/536312?v=3","gravatar_id":"","url":"https://api.github.com/users/otoolep","html_url":"https://github.com/otoolep","followers_url":"https://api.github.com/users/otoolep/followers","following_url":"https://api.github.com/users/otoolep/following{/other_user}","gists_url":"https://api.github.com/users/otoolep/gists{/gist_id}","starred_url":"https://api.github.com/users/otoolep/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/otoolep/subscriptions","organizations_url":"https://api.github.com/users/otoolep/orgs","repos_url":"https://api.github.com/users/otoolep/repos","events_url":"https://api.github.com/users/otoolep/events{/privacy}","received_events_url":"https://api.github.com/users/otoolep/received_events","type":"User","site_admin":false},"created_at":"2015-10-27T21:42:52Z","updated_at":"2015-10-27T21:42:52Z","body":"Use of `closing` right now should be OK, future enhancements may come."}}