{"action":"created","issue":{"url":"https://api.github.com/repos/shazow/urllib3/issues/743","labels_url":"https://api.github.com/repos/shazow/urllib3/issues/743/labels{/name}","comments_url":"https://api.github.com/repos/shazow/urllib3/issues/743/comments","events_url":"https://api.github.com/repos/shazow/urllib3/issues/743/events","html_url":"https://github.com/shazow/urllib3/issues/743","id":117657737,"number":743,"title":"Could calling  self._decoder.decompress after flush() has been called on the decompressobj instance cause some problems ?","user":{"login":"Vaudesir","id":3124194,"avatar_url":"https://avatars.githubusercontent.com/u/3124194?v=3","gravatar_id":"","url":"https://api.github.com/users/Vaudesir","html_url":"https://github.com/Vaudesir","followers_url":"https://api.github.com/users/Vaudesir/followers","following_url":"https://api.github.com/users/Vaudesir/following{/other_user}","gists_url":"https://api.github.com/users/Vaudesir/gists{/gist_id}","starred_url":"https://api.github.com/users/Vaudesir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Vaudesir/subscriptions","organizations_url":"https://api.github.com/users/Vaudesir/orgs","repos_url":"https://api.github.com/users/Vaudesir/repos","events_url":"https://api.github.com/users/Vaudesir/events{/privacy}","received_events_url":"https://api.github.com/users/Vaudesir/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-11-18T19:08:30Z","updated_at":"2015-11-20T08:11:26Z","closed_at":null,"body":"Disclaimer : I don’t know much about python. I am calling, from Java (through jython), a python script that uses a 3rd party python API that makes use of urllib3 to call a REST api. This whole setup obviously makes it challenging to troubleshoot issues.\r\nThe 3rd party python API we’re using uses urllib3 version 1.12\r\nWhile I know it would be very useful, I do not have a small sample to replicate the issue (I’d have to peel a lot of layers that I do not have a good understanding of at this point). I might need to work on it if nobody can help me without it.\r\n\r\nThe issue I am having is thrown by the zlib java implementation that is called by jython:\r\n\r\n```\r\njava.lang.NullPointerException: java.lang.NullPointerException: Inflater has been closed\r\n\r\nFile \"C:\\Users\\laurentd\\.gradle\\caches\\modules-2\\files-2.1\\org.python\\jython-standalone\\2.7.0\\cdfb38bc6f8343bcf1d6accc2e1147e8e7b63b75\\jython-standalone-2.7.0.jar\\Lib\\zlib.py\", line 202, in _get_inflate_data\r\n\tat java.util.zip.Inflater.ensureOpen(Inflater.java:389)\r\n\tat java.util.zip.Inflater.inflate(Inflater.java:257)\r\n\tat java.util.zip.Inflater.inflate(Inflater.java:280)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n```\r\n\r\nThe issue seems to be linked to the fact that the flush() method is called on a zlip.decompressobj object but that somehow, somewhere, we’re still trying to call the decompress method on that zlibdecompressobj object.\r\nThis is something that the zlib documentation mentions should not be done:\r\n“After calling flush(), the decompress() method cannot be called again; the only realistic action is to delete the object.” (from https://docs.python.org/2/library/zlib.html)\r\n\r\n\r\nFurther debugging took me to the response.py file, in the read_chunked method, especially in the last line of this part:\r\n```\r\n            while True:\r\n                self._update_chunk_length()\r\n                if self.chunk_left == 0:\r\n                    break\r\n                chunk = self._handle_chunk(amt)\r\n                yield self._decode(chunk, decode_content=decode_content,\r\n                                   flush_decoder=True)\r\n```\r\nIt looks like when self._decode is called, we always set flush_decoder to True.\r\n\r\nBecause of this, if I understand correctly, the self._decoder object (which is, in my case, a GzipDecoder, and a decompressobj?) is always flushed at the end of the call to self._decode (line 200).\r\nHowever, the self._decoder seems to be initialized only once and seems to be reused (at least in my case) throughout the multiple calls to self._decode.\r\n\r\nThough I am probably missing a few things, it could explain why decompressobj.decompress is called on a decompressobj object that has been flushed, and this could (?) have unpredictable results, including that NPE in the jython implementation of zlib (and ultimately in java).\r\n\r\nHere is a deeper (and sanitized) portion of the call stack when the exception happens.\r\n```\r\n....\\scripts\\requests\\sessions.py\", line 608, in send\r\n    r.content\r\n....\\scripts\\requests\\models.py\", line 734, in content\r\n    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()\r\n....\\scripts\\requests\\models.py\", line 734, in content\r\n    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()\r\n....\\scripts\\requests\\models.py\", line 657, in generate\r\n    for chunk in self.raw.stream(chunk_size, decode_content=True):\r\n....\\scripts\\requests\\models.py\", line 657, in generate\r\n    for chunk in self.raw.stream(chunk_size, decode_content=True):\r\n....\\scripts\\requests\\packages\\urllib3\\response.py\", line 322, in stream\r\n    for line in self.read_chunked(amt, decode_content=decode_content):\r\n....\\scripts\\requests\\packages\\urllib3\\response.py\", line 471, in read_chunked\r\n    yield self._decode(chunk, decode_content=decode_content,\r\n....\\scripts\\requests\\packages\\urllib3\\response.py\", line 190, in _decode\r\n    data = self._decoder.decompress(data)\r\n....\\scripts\\requests\\packages\\urllib3\\response.py\", line 190, in _decode\r\n    data = self._decoder.decompress(data)\r\n....\\scripts\\requests\\packages\\urllib3\\response.py\", line 56, in decompress\r\n    return self._obj.decompress(data)\r\n  File \"C:\\Users\\laurentd\\.gradle\\caches\\modules-2\\files-2.1\\org.python\\jython-standalone\\2.7.0\\cdfb38bc6f8343bcf1d6accc2e1147e8e7b63b75\\jython-standalone-2.7.0.jar\\Lib\\zlib.py\", line 155, in decompress\r\n  File \"C:\\Users\\laurentd\\.gradle\\caches\\modules-2\\files-2.1\\org.python\\jython-standalone\\2.7.0\\cdfb38bc6f8343bcf1d6accc2e1147e8e7b63b75\\jython-standalone-2.7.0.jar\\Lib\\zlib.py\", line 202, in _get_inflate_data\r\n  File \"C:\\Users\\laurentd\\.gradle\\caches\\modules-2\\files-2.1\\org.python\\jython-standalone\\2.7.0\\cdfb38bc6f8343bcf1d6accc2e1147e8e7b63b75\\jython-standalone-2.7.0.jar\\Lib\\zlib.py\", line 202, in _get_inflate_data\r\n```\r\nIs it possible that the _decode method is causing the problem by calling decompress on a decompressobj that has been flushed or am I going down the wrong track?\r\n\r\nNote that the problem is gone if I reinitialize the decompressobj at the beginning of _decode by mimicking what _init_decoder does, which I can’t believe is a good fix (although self._decoder seems to be really used only in _decode):\r\n```\r\n        content_encoding = self.headers.get('content-encoding', '').lower()\r\n        self._decoder = _get_decoder(content_encoding)\r\n```\r\n\r\nThanks in advance.\r\n\r\nLaurent.\r\n\r\n\r\n<bountysource-plugin>\r\n---\r\nWant to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/28367602-could-calling-self-_decoder-decompress-after-flush-has-been-called-on-the-decompressobj-instance-cause-some-problems?utm_campaign=plugin&utm_content=tracker%2F192525&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F192525&utm_medium=issues&utm_source=github).\r\n</bountysource-plugin>"},"comment":{"url":"https://api.github.com/repos/shazow/urllib3/issues/comments/158317532","html_url":"https://github.com/shazow/urllib3/issues/743#issuecomment-158317532","issue_url":"https://api.github.com/repos/shazow/urllib3/issues/743","id":158317532,"user":{"login":"Lukasa","id":1382556,"avatar_url":"https://avatars.githubusercontent.com/u/1382556?v=3","gravatar_id":"","url":"https://api.github.com/users/Lukasa","html_url":"https://github.com/Lukasa","followers_url":"https://api.github.com/users/Lukasa/followers","following_url":"https://api.github.com/users/Lukasa/following{/other_user}","gists_url":"https://api.github.com/users/Lukasa/gists{/gist_id}","starred_url":"https://api.github.com/users/Lukasa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Lukasa/subscriptions","organizations_url":"https://api.github.com/users/Lukasa/orgs","repos_url":"https://api.github.com/users/Lukasa/repos","events_url":"https://api.github.com/users/Lukasa/events{/privacy}","received_events_url":"https://api.github.com/users/Lukasa/received_events","type":"User","site_admin":false},"created_at":"2015-11-20T08:11:26Z","updated_at":"2015-11-20T08:11:26Z","body":"@sigmavirus24 I mean, I *suppose*? It's hard to see how useful that bug report would be for the Jython developers, really. The most that the bug could say is:\r\n\r\n1. You haven't exported some constants that exist in the CPython zlib library.\r\n2. You use zlib differently to CPython/PyPy, but totally within the bounds of what the docs say is allowed.\r\n\r\nPoint (1) is vaguely reasonable, point (2) is less clear. The docs certainly allow their current behaviour. That said, it can't hurt to at least tell them. =)"}}