{"action":"created","issue":{"url":"https://api.github.com/repos/Parsely/pykafka/issues/379","labels_url":"https://api.github.com/repos/Parsely/pykafka/issues/379/labels{/name}","comments_url":"https://api.github.com/repos/Parsely/pykafka/issues/379/comments","events_url":"https://api.github.com/repos/Parsely/pykafka/issues/379/events","html_url":"https://github.com/Parsely/pykafka/issues/379","id":120580642,"number":379,"title":"rdkafka memory issue","user":{"login":"jofusa","id":800669,"avatar_url":"https://avatars.githubusercontent.com/u/800669?v=3","gravatar_id":"","url":"https://api.github.com/users/jofusa","html_url":"https://github.com/jofusa","followers_url":"https://api.github.com/users/jofusa/followers","following_url":"https://api.github.com/users/jofusa/following{/other_user}","gists_url":"https://api.github.com/users/jofusa/gists{/gist_id}","starred_url":"https://api.github.com/users/jofusa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jofusa/subscriptions","organizations_url":"https://api.github.com/users/jofusa/orgs","repos_url":"https://api.github.com/users/jofusa/repos","events_url":"https://api.github.com/users/jofusa/events{/privacy}","received_events_url":"https://api.github.com/users/jofusa/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Parsely/pykafka/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/Parsely/pykafka/labels/in%20progress","name":"in progress","color":"ededed"}],"state":"open","locked":false,"assignee":{"login":"yungchin","id":780001,"avatar_url":"https://avatars.githubusercontent.com/u/780001?v=3","gravatar_id":"","url":"https://api.github.com/users/yungchin","html_url":"https://github.com/yungchin","followers_url":"https://api.github.com/users/yungchin/followers","following_url":"https://api.github.com/users/yungchin/following{/other_user}","gists_url":"https://api.github.com/users/yungchin/gists{/gist_id}","starred_url":"https://api.github.com/users/yungchin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yungchin/subscriptions","organizations_url":"https://api.github.com/users/yungchin/orgs","repos_url":"https://api.github.com/users/yungchin/repos","events_url":"https://api.github.com/users/yungchin/events{/privacy}","received_events_url":"https://api.github.com/users/yungchin/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/Parsely/pykafka/milestones/12","html_url":"https://github.com/Parsely/pykafka/milestones/2.1.1","labels_url":"https://api.github.com/repos/Parsely/pykafka/milestones/12/labels","id":1438246,"number":12,"title":"2.1.1","description":"","creator":{"login":"emmett9001","id":723615,"avatar_url":"https://avatars.githubusercontent.com/u/723615?v=3","gravatar_id":"","url":"https://api.github.com/users/emmett9001","html_url":"https://github.com/emmett9001","followers_url":"https://api.github.com/users/emmett9001/followers","following_url":"https://api.github.com/users/emmett9001/following{/other_user}","gists_url":"https://api.github.com/users/emmett9001/gists{/gist_id}","starred_url":"https://api.github.com/users/emmett9001/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emmett9001/subscriptions","organizations_url":"https://api.github.com/users/emmett9001/orgs","repos_url":"https://api.github.com/users/emmett9001/repos","events_url":"https://api.github.com/users/emmett9001/events{/privacy}","received_events_url":"https://api.github.com/users/emmett9001/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":2,"state":"open","created_at":"2015-12-01T20:02:59Z","updated_at":"2015-12-06T05:07:32Z","due_on":null,"closed_at":null},"comments":5,"created_at":"2015-12-05T19:10:31Z","updated_at":"2015-12-06T18:15:34Z","closed_at":null,"body":"First wonderful project, i've watching from the sidelines and really excited about pushing for first class python tools in the kafka ecosystem. \r\n\r\nI have started testing the new rdkafka support in 2.1.0 and i do see a throughput improvement (yeah!), but there appears to be a memory leak. My attempts to implement memory profiling have come up with very little because the tooling around C extensions is limited and my knowledge is even less. \r\n\r\nI'm going to dig deeper next week but wanted to put a note here to see if anyone else run into this. \r\n\r\nI am deploying these workers as docker containers, so i am not ruling out some issue with my container setup, but below is a chart of memory.rss for the same container. The flat lines are configured `use_rdkafka=False`, the ominous slanted line only difference is `use_rdkafka=True`. These are configured from envvars so the underlying images are the same.\r\n\r\n<img width=\"603\" alt=\"screen shot 2015-12-05 at 12 02 20 pm\" src=\"https://cloud.githubusercontent.com/assets/800669/11609593/2f9a0642-9b48-11e5-8f86-ea8b3ea0b8af.png\">\r\n\r\nSorry for the vagueness of this, i work on getting some better threads to pull on next week.\r\n\r\nMy setup:\r\npykafka 2.1.0\r\nrdkafka 0.8.6"},"comment":{"url":"https://api.github.com/repos/Parsely/pykafka/issues/comments/162334094","html_url":"https://github.com/Parsely/pykafka/issues/379#issuecomment-162334094","issue_url":"https://api.github.com/repos/Parsely/pykafka/issues/379","id":162334094,"user":{"login":"jofusa","id":800669,"avatar_url":"https://avatars.githubusercontent.com/u/800669?v=3","gravatar_id":"","url":"https://api.github.com/users/jofusa","html_url":"https://github.com/jofusa","followers_url":"https://api.github.com/users/jofusa/followers","following_url":"https://api.github.com/users/jofusa/following{/other_user}","gists_url":"https://api.github.com/users/jofusa/gists{/gist_id}","starred_url":"https://api.github.com/users/jofusa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jofusa/subscriptions","organizations_url":"https://api.github.com/users/jofusa/orgs","repos_url":"https://api.github.com/users/jofusa/repos","events_url":"https://api.github.com/users/jofusa/events{/privacy}","received_events_url":"https://api.github.com/users/jofusa/received_events","type":"User","site_admin":false},"created_at":"2015-12-06T18:15:34Z","updated_at":"2015-12-06T18:15:34Z","body":"Thanks for fixing! thats a nasty one. who said C extensions were hard...\r\n\r\nI tried this with no luck. https://pythonhosted.org/Pympler/muppy.html\r\n\r\nAt some point, aggregate cpu hours are what is needed. Which means RCs and a willing community."}}