{"action":"created","issue":{"url":"https://api.github.com/repos/joblib/joblib/issues/260","labels_url":"https://api.github.com/repos/joblib/joblib/issues/260/labels{/name}","comments_url":"https://api.github.com/repos/joblib/joblib/issues/260/comments","events_url":"https://api.github.com/repos/joblib/joblib/issues/260/events","html_url":"https://github.com/joblib/joblib/pull/260","id":113053476,"number":260,"title":"[MRG] Numpy pickle to single file","user":{"login":"aabadie","id":1375137,"avatar_url":"https://avatars.githubusercontent.com/u/1375137?v=3","gravatar_id":"","url":"https://api.github.com/users/aabadie","html_url":"https://github.com/aabadie","followers_url":"https://api.github.com/users/aabadie/followers","following_url":"https://api.github.com/users/aabadie/following{/other_user}","gists_url":"https://api.github.com/users/aabadie/gists{/gist_id}","starred_url":"https://api.github.com/users/aabadie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aabadie/subscriptions","organizations_url":"https://api.github.com/users/aabadie/orgs","repos_url":"https://api.github.com/users/aabadie/repos","events_url":"https://api.github.com/users/aabadie/events{/privacy}","received_events_url":"https://api.github.com/users/aabadie/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":50,"created_at":"2015-10-23T16:22:17Z","updated_at":"2015-11-23T15:25:56Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/joblib/joblib/pulls/260","html_url":"https://github.com/joblib/joblib/pull/260","diff_url":"https://github.com/joblib/joblib/pull/260.diff","patch_url":"https://github.com/joblib/joblib/pull/260.patch"},"body":"I gave a try to gzip.GzipFile to dump numpy object. This PR is a POC of such usage and also contains a fairly large refactoring of numpy_pickle.\r\n\r\nIn terms of performance, I reused the bench script from #255. With Numpy 1.10, there's no memory copy but the read/write is slightly slower (31s dump, 5.5s read). It also makes the code more 'readable'.\r\nI had to comment out the test regarding the compatibility between pickles.\r\n\r\nWaiting for the CI's"},"comment":{"url":"https://api.github.com/repos/joblib/joblib/issues/comments/158967641","html_url":"https://github.com/joblib/joblib/pull/260#issuecomment-158967641","issue_url":"https://api.github.com/repos/joblib/joblib/issues/260","id":158967641,"user":{"login":"aabadie","id":1375137,"avatar_url":"https://avatars.githubusercontent.com/u/1375137?v=3","gravatar_id":"","url":"https://api.github.com/users/aabadie","html_url":"https://github.com/aabadie","followers_url":"https://api.github.com/users/aabadie/followers","following_url":"https://api.github.com/users/aabadie/following{/other_user}","gists_url":"https://api.github.com/users/aabadie/gists{/gist_id}","starred_url":"https://api.github.com/users/aabadie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aabadie/subscriptions","organizations_url":"https://api.github.com/users/aabadie/orgs","repos_url":"https://api.github.com/users/aabadie/repos","events_url":"https://api.github.com/users/aabadie/events{/privacy}","received_events_url":"https://api.github.com/users/aabadie/received_events","type":"User","site_admin":false},"created_at":"2015-11-23T15:25:56Z","updated_at":"2015-11-23T15:25:56Z","body":"Sorry, I was busy this afternoon. \r\n\r\n> I would still like to see the benchmark scripts with CRC checking disabled at write and at read.\r\n\r\nI'll post new comparison charts between both gzip (with and without crc32) and master.\r\n\r\n> it seems that removing CRCs could gain something like 30% of the time.\r\n\r\nThat's I could observe.\r\n\r\n> Quickly looking at the gzip code, one way to disable the CRC would be to derive from gzip.GzipFile and override _add_read_data (to not do the CRC computation) and _read_eof (to not do the CRC check).\r\n\r\nI also thought of this solution as I remember numpy was doing the same for the gzip seek function."}}