{"action":"created","issue":{"url":"https://api.github.com/repos/KeepSafe/aiohttp/issues/271","labels_url":"https://api.github.com/repos/KeepSafe/aiohttp/issues/271/labels{/name}","comments_url":"https://api.github.com/repos/KeepSafe/aiohttp/issues/271/comments","events_url":"https://api.github.com/repos/KeepSafe/aiohttp/issues/271/events","html_url":"https://github.com/KeepSafe/aiohttp/issues/271","id":56894439,"number":271,"title":"Memory leak in request","user":{"login":"mpaolini","id":523292,"avatar_url":"https://avatars.githubusercontent.com/u/523292?v=3","gravatar_id":"","url":"https://api.github.com/users/mpaolini","html_url":"https://github.com/mpaolini","followers_url":"https://api.github.com/users/mpaolini/followers","following_url":"https://api.github.com/users/mpaolini/following{/other_user}","gists_url":"https://api.github.com/users/mpaolini/gists{/gist_id}","starred_url":"https://api.github.com/users/mpaolini/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mpaolini/subscriptions","organizations_url":"https://api.github.com/users/mpaolini/orgs","repos_url":"https://api.github.com/users/mpaolini/repos","events_url":"https://api.github.com/users/mpaolini/events{/privacy}","received_events_url":"https://api.github.com/users/mpaolini/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-02-07T02:54:25Z","updated_at":"2015-03-11T17:05:42Z","closed_at":null,"body":"Hi all,\r\n\r\n Since I upgraded to 0.14.4 (from 0.9.0)  I am experiencing memory leaks in a Dropbox-API longpoller. It is a single process that spawns a few thousands of greenlets. Each greenlet performs a `request()`, that blocks for 30 seconds, then parses the response and dies. Then a new greenlet is spawned.\r\n\r\nI am running on python 3.4.0, Ubuntu 14.04. I use the connection pool feature, passing the same connector singleton to each `.request()` call.\r\n\r\nI played with tracemalloc, dumping a `<N>.dump` stat file every minute and found out that the response parser instances keep increasing in number (look at the third line of each stat)\r\n\r\n```\r\nroot@9330490eafc9:/src# python3 -c \"import pprint; import tracemalloc; pprint.pprint(tracemalloc.Snapshot.load('5.dump').statistics('lineno')[:3])\"\r\n[<Statistic traceback=<Traceback (<Frame filename='/usr/lib/python3.4/ssl.py' lineno=648>,)> size=6130540 count=82650>,\r\n <Statistic traceback=<Traceback (<Frame filename='<frozen importlib._bootstrap>' lineno=656>,)> size=3679906 count=31688>,\r\n <Statistic traceback=<Traceback (<Frame filename='/usr/local/lib/python3.4/dist-packages/aiohttp/parsers.py' lineno=198>,)> size=2176408 count=4437>]\r\nroot@9330490eafc9:/src# \r\nroot@9330490eafc9:/src# \r\nroot@9330490eafc9:/src# python3 -c \"import pprint; import tracemalloc; pprint.pprint(tracemalloc.Snapshot.load('6.dump').statistics('lineno')[:3])\"\r\n[<Statistic traceback=<Traceback (<Frame filename='/usr/lib/python3.4/ssl.py' lineno=648>,)> size=6130476 count=82649>,\r\n <Statistic traceback=<Traceback (<Frame filename='<frozen importlib._bootstrap>' lineno=656>,)> size=3679906 count=31688>,\r\n <Statistic traceback=<Traceback (<Frame filename='/usr/local/lib/python3.4/dist-packages/aiohttp/parsers.py' lineno=198>,)> size=2199704 count=4463>]\r\nroot@9330490eafc9:/src# \r\nroot@9330490eafc9:/src# \r\nroot@9330490eafc9:/src# python3 -c \"import pprint; import tracemalloc; pprint.pprint(tracemalloc.Snapshot.load('7.dump').statistics('lineno')[:3])\"\r\n[<Statistic traceback=<Traceback (<Frame filename='/usr/lib/python3.4/ssl.py' lineno=648>,)> size=6130476 count=82649>,\r\n <Statistic traceback=<Traceback (<Frame filename='<frozen importlib._bootstrap>' lineno=656>,)> size=3679906 count=31688>,\r\n <Statistic traceback=<Traceback (<Frame filename='/usr/local/lib/python3.4/dist-packages/aiohttp/parsers.py' lineno=198>,)> size=2231064 count=4498>]\r\n```\r\n\r\ntracemalloc reports this stack trace:\r\n\r\n```\r\npython3 -c \"import pprint; import tracemalloc; pprint.pprint(tracemalloc.Snapshot.load('3.dump').filter_traces([tracemalloc.Filter(True, '*aiohttp/parsers.py')]).statistics('traceback')[0].traceback.format())\"\r\n['  File \"/usr/local/lib/python3.4/dist-packages/aiohttp/parsers.py\", line '\r\n '198',\r\n '    p = parser(output, self._buffer)',\r\n '  File \"/usr/local/lib/python3.4/dist-packages/aiohttp/client.py\", line 633',\r\n '    httpstream = self._reader.set_parser(self._response_parser)',\r\n '  File \"/usr/local/lib/python3.4/dist-packages/aiohttp/client.py\", line 108',\r\n '    yield from resp.start(conn, read_until_eof)',\r\n '  File \"/src/xxxxxx/main.py\", line 70',\r\n '    connector=LONGPOLL_CONNECTOR',\r\n```\r\n\r\nLooks like there is something keeping alive those parsers....\r\n\r\nUsing `force_close=True` on the connector makes no difference.\r\n\r\nThen I tried calling `gc.collect()` after every single request, and is going much better ~~but the leak has not~~ the leak **has** disappeared completely-. This means (maybe is an unrelated issue) the library creates more reference cycles thant the CGC can handle.\r\n\r\nIt my well be my own bug, or maybe something to do with python 3.4.0 itself. I'm still digging into it.\r\n"},"comment":{"url":"https://api.github.com/repos/KeepSafe/aiohttp/issues/comments/78307695","html_url":"https://github.com/KeepSafe/aiohttp/issues/271#issuecomment-78307695","issue_url":"https://api.github.com/repos/KeepSafe/aiohttp/issues/271","id":78307695,"user":{"login":"fafhrd91","id":598990,"avatar_url":"https://avatars.githubusercontent.com/u/598990?v=3","gravatar_id":"","url":"https://api.github.com/users/fafhrd91","html_url":"https://github.com/fafhrd91","followers_url":"https://api.github.com/users/fafhrd91/followers","following_url":"https://api.github.com/users/fafhrd91/following{/other_user}","gists_url":"https://api.github.com/users/fafhrd91/gists{/gist_id}","starred_url":"https://api.github.com/users/fafhrd91/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fafhrd91/subscriptions","organizations_url":"https://api.github.com/users/fafhrd91/orgs","repos_url":"https://api.github.com/users/fafhrd91/repos","events_url":"https://api.github.com/users/fafhrd91/events{/privacy}","received_events_url":"https://api.github.com/users/fafhrd91/received_events","type":"User","site_admin":false},"created_at":"2015-03-11T17:05:42Z","updated_at":"2015-03-11T17:05:42Z","body":"@mpaolini could you try latest master?"}}