{"action":"opened","issue":{"url":"https://api.github.com/repos/blaze/dask/issues/888","labels_url":"https://api.github.com/repos/blaze/dask/issues/888/labels{/name}","comments_url":"https://api.github.com/repos/blaze/dask/issues/888/comments","events_url":"https://api.github.com/repos/blaze/dask/issues/888/events","html_url":"https://github.com/blaze/dask/issues/888","id":123580009,"number":888,"title":"Unstable dask._Frame.map_partitions behavior","user":{"login":"sscondie","id":328577,"avatar_url":"https://avatars.githubusercontent.com/u/328577?v=3","gravatar_id":"","url":"https://api.github.com/users/sscondie","html_url":"https://github.com/sscondie","followers_url":"https://api.github.com/users/sscondie/followers","following_url":"https://api.github.com/users/sscondie/following{/other_user}","gists_url":"https://api.github.com/users/sscondie/gists{/gist_id}","starred_url":"https://api.github.com/users/sscondie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sscondie/subscriptions","organizations_url":"https://api.github.com/users/sscondie/orgs","repos_url":"https://api.github.com/users/sscondie/repos","events_url":"https://api.github.com/users/sscondie/events{/privacy}","received_events_url":"https://api.github.com/users/sscondie/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-12-23T00:16:56Z","updated_at":"2015-12-23T00:16:56Z","closed_at":null,"body":"Consider the following script operating on 15 csv files each about 16MB in size (each csv is an array of floats that is 10 columns and 100,000 rows).  \r\n\r\nimport dask.dataframe as dd\r\ndf = dd.read_csv(\"test_data_small/*.csv\")\r\nprint(df.npartitions)\r\nprint(df.mean().compute())\r\nprint(df.map_partitions(len,columns=df.columns).compute())\r\n\r\nRunning this script on my machine (OS X (10.11.1), Python 2.7.11 [Anaconda 2.3.0 (x86_64)], pandas 0.17.0) produces (in approximately the same frequencies and in random order) one of the following results (for each result I give first a description and then stdout)\r\n\r\n\r\n1.  npartitions completes, mean completes, map_partitions seg faults\r\n\r\n15\r\n0                -0.000135\r\n1                -0.000011\r\n10               -0.000939\r\n2                 0.000771\r\n3                 0.000309\r\n4                 0.000573\r\n5                 0.000635\r\n6                 0.000770\r\n7                 0.000713\r\n8                 0.001237\r\n9                -0.000440\r\nUnnamed: 0    49999.500000\r\ndtype: float64\r\nSegmentation fault: 11\r\n\r\n2. npartitions completes, Fatal Python Error at mean \r\n\r\n15\r\nFatal Python error: GC object already tracked\r\nAbort trap: 6\r\n\r\n3. npartitions completes, mean completes, Fatal Python Error at map_partitions\r\n \r\n15\r\n0                -0.000135\r\n1                -0.000011\r\n10               -0.000939\r\n2                 0.000771\r\n3                 0.000309\r\n4                 0.000573\r\n5                 0.000635\r\n6                 0.000770\r\n7                 0.000713\r\n8                 0.001237\r\n9                -0.000440\r\nUnnamed: 0    49999.500000\r\ndtype: float64\r\nFatal Python error: GC object already tracked\r\nAbort trap: 6\r\n\r\n4.  npartitions completes, seg fault at mean\r\n\r\n15\r\nSegmentation fault: 11\r\n\r\n5.  Everything completes\r\n\r\n15\r\n0                -0.000135\r\n1                -0.000011\r\n10               -0.000939\r\n2                 0.000771\r\n3                 0.000309\r\n4                 0.000573\r\n5                 0.000635\r\n6                 0.000770\r\n7                 0.000713\r\n8                 0.001237\r\n9                -0.000440\r\nUnnamed: 0    49999.500000\r\ndtype: float64\r\n(100001, 100001, 100001, 100001, 100001, 100001, 100001, 100001, 100001, 100001, 100001, 100001, 100001, 100001, 100001)\r\n\r\nI reverted pandas back to 0.17.0 based on a suggestion from Matt [here] http://stackoverflow.com/questions/34128540/why-is-running-a-compute-in-dask-causing-fatal-python-error-gc-object-alrea  but that hasn't noticeably changed the results.\r\n\r\nI'm happy to debug this in dask, but there is at least some chance that this could be a pandas problem.  Has anyone else seen this kind of behavior?  If so, were you able to find an answer?\r\n\r\n"}}