{"action":"created","issue":{"url":"https://api.github.com/repos/blaze/dask/issues/865","labels_url":"https://api.github.com/repos/blaze/dask/issues/865/labels{/name}","comments_url":"https://api.github.com/repos/blaze/dask/issues/865/comments","events_url":"https://api.github.com/repos/blaze/dask/issues/865/events","html_url":"https://github.com/blaze/dask/pull/865","id":120590443,"number":865,"title":"Replace dill with cloudpickle","user":{"login":"PedroMDuarte","id":921258,"avatar_url":"https://avatars.githubusercontent.com/u/921258?v=3","gravatar_id":"","url":"https://api.github.com/users/PedroMDuarte","html_url":"https://github.com/PedroMDuarte","followers_url":"https://api.github.com/users/PedroMDuarte/followers","following_url":"https://api.github.com/users/PedroMDuarte/following{/other_user}","gists_url":"https://api.github.com/users/PedroMDuarte/gists{/gist_id}","starred_url":"https://api.github.com/users/PedroMDuarte/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PedroMDuarte/subscriptions","organizations_url":"https://api.github.com/users/PedroMDuarte/orgs","repos_url":"https://api.github.com/users/PedroMDuarte/repos","events_url":"https://api.github.com/users/PedroMDuarte/events{/privacy}","received_events_url":"https://api.github.com/users/PedroMDuarte/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-12-05T21:56:06Z","updated_at":"2015-12-09T15:41:56Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/blaze/dask/pulls/865","html_url":"https://github.com/blaze/dask/pull/865","diff_url":"https://github.com/blaze/dask/pull/865.diff","patch_url":"https://github.com/blaze/dask/pull/865.patch"},"body":"WIP for #847 \r\n\r\n* updated docs, travis, and conda.recipe to use `cloudpickle`.  \r\n\r\n* left dill in the travis installs because `test_dataframe_series_are_dillable` uses it\r\n\r\n* centralized the choice between `dill` and `cloudpickle` at `dask/multiprocessing.py`.   Other tests where `dill` was imported now use `multiprocessing._dumps` and `multiprocessing._loads`. \r\n\r\n* added a test for pickling functions with `__globals__`: `dask/tests/test_multiprocessing.py::test_pickle_globals`. \r\n    - The test fails with `dill` and passes with `cloudpickle`\r\n\r\n* the switch to `cloudpickle` introduced regressions in some tests for Python <= 3.3\r\n    - `dask/bag/core.py::dask.bag.core.Bag.distinct FAILED`  (failed in Python:2.7 only)\r\n    - `dask/bag/tests/test_bag.py::test_fold FAILED`\r\n    - `dask/bag/tests/test_bag.py::test_distinct FAILED`\r\n    - `dask/bag/tests/test_bag.py::test_from_filenames_large FAILED `\r\n    - `dask/bag/tests/test_bag.py::test_from_filenames_encoding FAILED`\r\n    - `dask/bag/tests/test_bag.py::test_from_filenames_large_gzip FAILED`\r\n\r\n* the error seen in the FAILED tests above was always: `PicklingError: Could not pickle object as excessively deep recursion required.`\r\n* I introduced a somewhat ugly hack in `dask/multiprocessing.py` that fixes the recursion errors seen so far:  if `dumps` fails the recursion limit is increased by a factor of 2 and `cloudpickle` tries again. \r\n\r\nI appreciate any feedback on this WIP.   Thanks! \r\n"},"comment":{"url":"https://api.github.com/repos/blaze/dask/issues/comments/163296920","html_url":"https://github.com/blaze/dask/pull/865#issuecomment-163296920","issue_url":"https://api.github.com/repos/blaze/dask/issues/865","id":163296920,"user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars.githubusercontent.com/u/306380?v=3","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2015-12-09T15:41:56Z","updated_at":"2015-12-09T15:41:56Z","body":"Looks good to me.  Merging in 12 hours if no comments"}}