{"action":"created","issue":{"url":"https://api.github.com/repos/joblib/joblib/issues/260","labels_url":"https://api.github.com/repos/joblib/joblib/issues/260/labels{/name}","comments_url":"https://api.github.com/repos/joblib/joblib/issues/260/comments","events_url":"https://api.github.com/repos/joblib/joblib/issues/260/events","html_url":"https://github.com/joblib/joblib/pull/260","id":113053476,"number":260,"title":"[WIP] Gzip pickling","user":{"login":"aabadie","id":1375137,"avatar_url":"https://avatars.githubusercontent.com/u/1375137?v=3","gravatar_id":"","url":"https://api.github.com/users/aabadie","html_url":"https://github.com/aabadie","followers_url":"https://api.github.com/users/aabadie/followers","following_url":"https://api.github.com/users/aabadie/following{/other_user}","gists_url":"https://api.github.com/users/aabadie/gists{/gist_id}","starred_url":"https://api.github.com/users/aabadie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aabadie/subscriptions","organizations_url":"https://api.github.com/users/aabadie/orgs","repos_url":"https://api.github.com/users/aabadie/repos","events_url":"https://api.github.com/users/aabadie/events{/privacy}","received_events_url":"https://api.github.com/users/aabadie/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-10-23T16:22:17Z","updated_at":"2015-11-12T16:54:01Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/joblib/joblib/pulls/260","html_url":"https://github.com/joblib/joblib/pull/260","diff_url":"https://github.com/joblib/joblib/pull/260.diff","patch_url":"https://github.com/joblib/joblib/pull/260.patch"},"body":"I gave a try to gzip.GzipFile to dump numpy object. This PR is a POC of such usage and also contains a fairly large refactoring of numpy_pickle.\r\n\r\nIn terms of performance, I reused the bench script from #255. With Numpy 1.10, there's no memory copy but the read/write is slightly slower (31s dump, 5.5s read). It also makes the code more 'readable'.\r\nI had to comment out the test regarding the compatibility between pickles.\r\n\r\nWaiting for the CI's"},"comment":{"url":"https://api.github.com/repos/joblib/joblib/issues/comments/156164260","html_url":"https://github.com/joblib/joblib/pull/260#issuecomment-156164260","issue_url":"https://api.github.com/repos/joblib/joblib/issues/260","id":156164260,"user":{"login":"aabadie","id":1375137,"avatar_url":"https://avatars.githubusercontent.com/u/1375137?v=3","gravatar_id":"","url":"https://api.github.com/users/aabadie","html_url":"https://github.com/aabadie","followers_url":"https://api.github.com/users/aabadie/followers","following_url":"https://api.github.com/users/aabadie/following{/other_user}","gists_url":"https://api.github.com/users/aabadie/gists{/gist_id}","starred_url":"https://api.github.com/users/aabadie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aabadie/subscriptions","organizations_url":"https://api.github.com/users/aabadie/orgs","repos_url":"https://api.github.com/users/aabadie/repos","events_url":"https://api.github.com/users/aabadie/events{/privacy}","received_events_url":"https://api.github.com/users/aabadie/received_events","type":"User","site_admin":false},"created_at":"2015-11-12T16:54:01Z","updated_at":"2015-11-12T16:54:01Z","body":"> the control file (containing the main serialized object without arrays/matrices) is not compressed\r\n\r\nThis was done in 4f58113\r\n\r\n> the whole object (including arrays) could be serialized in a single file using a smart seeking in the file. This could be part of an extra PR.\r\n\r\nI just pushed caeb704 which does this. But it has several drawbacks:\r\n* tests are broken because the actual state doesn't support memory mapping\r\n* I had to patch numpy locally in order to make the gzip file compression work. Here, numpy overload `GzipFile` object `seek()` function in order to be able to seek using negative values. The problem is that numpy assumes the file only contains one array which is not our case here. The only solution is to include numpy array load/save implementation in joblib.\r\n\r\n> last, but not least : this PR should also contains a retrocompatibility mecanism with previous versions of the cache format.\r\n\r\nThis is done in this branch (works in 4f58113)\r\n"}}