{"action":"created","issue":{"url":"https://api.github.com/repos/uber/tchannel-node/issues/83","labels_url":"https://api.github.com/repos/uber/tchannel-node/issues/83/labels{/name}","comments_url":"https://api.github.com/repos/uber/tchannel-node/issues/83/comments","events_url":"https://api.github.com/repos/uber/tchannel-node/issues/83/events","html_url":"https://github.com/uber/tchannel-node/issues/83","id":111495850,"number":83,"title":"Memory usage spikes after connections get in bad state","user":{"login":"markyen","id":673688,"avatar_url":"https://avatars.githubusercontent.com/u/673688?v=3","gravatar_id":"","url":"https://api.github.com/users/markyen","html_url":"https://github.com/markyen","followers_url":"https://api.github.com/users/markyen/followers","following_url":"https://api.github.com/users/markyen/following{/other_user}","gists_url":"https://api.github.com/users/markyen/gists{/gist_id}","starred_url":"https://api.github.com/users/markyen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markyen/subscriptions","organizations_url":"https://api.github.com/users/markyen/orgs","repos_url":"https://api.github.com/users/markyen/repos","events_url":"https://api.github.com/users/markyen/events{/privacy}","received_events_url":"https://api.github.com/users/markyen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-10-14T21:19:39Z","updated_at":"2015-10-23T19:15:10Z","closed_at":null,"body":"After the error in uber/hyperbahn#22 occurs, the memory on several workers spike to several gigabytes:\r\n\r\n![image](https://cloud.githubusercontent.com/assets/673688/10497634/c1d1d8f0-727b-11e5-9cfc-d5d5b8e203ef.png)\r\n\r\nA summary of core dumps taken from affected and normal workers are here: https://gist.github.com/markyen/67286b06b68d1b0b7685\r\n\r\nTODO:\r\n- [ ] Reap out requests in 30 second reaper\r\n- [ ] Remove arg2 and arg3 from req/res\r\n- [ ] Verify operations after error frame\r\n- [ ] Suspect that all out requests have an err field that is a timeout\r\n- [ ] Write to socket after close\r\n- [ ] Search for \"Writable\" or \"Duplex\"\r\n- [ ] Must purge connection when reset\r\n- [ ] Forcibly clear socket write queue\r\n- [ ] Remove closed connections from memory\r\n- [ ] Stale connections\r\n- [ ] write helper in hyperbahn for verifying empty operations \r\n- [ ] write test in hyperbahn for streaming call requests timing out.\r\n\r\n/cc @Raynos @ShanniLi "},"comment":{"url":"https://api.github.com/repos/uber/tchannel-node/issues/comments/150667174","html_url":"https://github.com/uber/tchannel-node/issues/83#issuecomment-150667174","issue_url":"https://api.github.com/repos/uber/tchannel-node/issues/83","id":150667174,"user":{"login":"Raynos","id":479538,"avatar_url":"https://avatars.githubusercontent.com/u/479538?v=3","gravatar_id":"","url":"https://api.github.com/users/Raynos","html_url":"https://github.com/Raynos","followers_url":"https://api.github.com/users/Raynos/followers","following_url":"https://api.github.com/users/Raynos/following{/other_user}","gists_url":"https://api.github.com/users/Raynos/gists{/gist_id}","starred_url":"https://api.github.com/users/Raynos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Raynos/subscriptions","organizations_url":"https://api.github.com/users/Raynos/orgs","repos_url":"https://api.github.com/users/Raynos/repos","events_url":"https://api.github.com/users/Raynos/events{/privacy}","received_events_url":"https://api.github.com/users/Raynos/received_events","type":"User","site_admin":false},"created_at":"2015-10-23T19:15:09Z","updated_at":"2015-10-23T19:15:09Z","body":"# Remove closed connections from memory\r\n\r\nOnce again we have closed connections, we never should have them lying around.\r\n\r\nLet's make sure they are really really gone and we are not leaking them.\r\n"}}