{"action":"created","issue":{"url":"https://api.github.com/repos/blaze/dask/issues/862","labels_url":"https://api.github.com/repos/blaze/dask/issues/862/labels{/name}","comments_url":"https://api.github.com/repos/blaze/dask/issues/862/comments","events_url":"https://api.github.com/repos/blaze/dask/issues/862/events","html_url":"https://github.com/blaze/dask/issues/862","id":120427081,"number":862,"title":"Error when slicing dask dataframes ","user":{"login":"fjanoos","id":923438,"avatar_url":"https://avatars.githubusercontent.com/u/923438?v=3","gravatar_id":"","url":"https://api.github.com/users/fjanoos","html_url":"https://github.com/fjanoos","followers_url":"https://api.github.com/users/fjanoos/followers","following_url":"https://api.github.com/users/fjanoos/following{/other_user}","gists_url":"https://api.github.com/users/fjanoos/gists{/gist_id}","starred_url":"https://api.github.com/users/fjanoos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fjanoos/subscriptions","organizations_url":"https://api.github.com/users/fjanoos/orgs","repos_url":"https://api.github.com/users/fjanoos/repos","events_url":"https://api.github.com/users/fjanoos/events{/privacy}","received_events_url":"https://api.github.com/users/fjanoos/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-12-04T15:41:38Z","updated_at":"2015-12-04T16:06:55Z","closed_at":null,"body":"Hello,\r\n\r\nThis is an error regarding the output of _loc_slice when slicing a dataframe that was read from a csv. \r\n\r\nThe problem seems to arise that with a csv, partition boundaries are not known before hand and I have to manually fix them as follows\r\n\r\n``` python\r\nimport dask.dataframe as ddf\r\nmy_ddf = ddf.read_csv('myfile.csv', chunkbytes=1000 ) # force multiple partitions\r\npartition_lens = my_ddf.map_partitions(len).compute() # get partition boundaries\r\nunk_divisions = [0]\r\nunk_divisions.extend(cumsum( partition_lens ))\r\nmy_ddf.divisions = tuple(unk_divisions)\r\n\r\nprint( my_ddf.divisions )\r\n```\r\n![image](https://cloud.githubusercontent.com/assets/923438/11593593/1b02109c-9a73-11e5-9121-d59a18d2fe6d.png)\r\n\r\n\r\n\r\nIssue 1: Slicing *within* a division breaks  the [inclusive, exclusive) semantics of python / pandas.\r\n\r\n```\r\nmy_ddf.loc[3:6].compute()\r\n```\r\n![image](https://cloud.githubusercontent.com/assets/923438/11593626/40cae092-9a73-11e5-96a0-2702e659bf18.png)\r\n\r\n\r\nIssue 2: Slicing *across* divisions (for a csv file read and fixed in this fashion) dumps the whole table and doesn't return the correct indices\r\n\r\n```\r\nmy_ddf.loc[8:11].compute()\r\n```\r\n![image](https://cloud.githubusercontent.com/assets/923438/11593648/5f6511e4-9a73-11e5-9b02-e223b75dd688.png)\r\n"},"comment":{"url":"https://api.github.com/repos/blaze/dask/issues/comments/162006423","html_url":"https://github.com/blaze/dask/issues/862#issuecomment-162006423","issue_url":"https://api.github.com/repos/blaze/dask/issues/862","id":162006423,"user":{"login":"fjanoos","id":923438,"avatar_url":"https://avatars.githubusercontent.com/u/923438?v=3","gravatar_id":"","url":"https://api.github.com/users/fjanoos","html_url":"https://github.com/fjanoos","followers_url":"https://api.github.com/users/fjanoos/followers","following_url":"https://api.github.com/users/fjanoos/following{/other_user}","gists_url":"https://api.github.com/users/fjanoos/gists{/gist_id}","starred_url":"https://api.github.com/users/fjanoos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fjanoos/subscriptions","organizations_url":"https://api.github.com/users/fjanoos/orgs","repos_url":"https://api.github.com/users/fjanoos/repos","events_url":"https://api.github.com/users/fjanoos/events{/privacy}","received_events_url":"https://api.github.com/users/fjanoos/received_events","type":"User","site_admin":false},"created_at":"2015-12-04T16:06:55Z","updated_at":"2015-12-04T16:06:55Z","body":"@mrocklin \r\n\r\nThe pandas  (version 0.17.0') I'm using has [, ) indexing:\r\n```\r\nmy_df = DataFrame( randn(1000,10))\r\nmy_df[3:5]\r\n```\r\n![image](https://cloud.githubusercontent.com/assets/923438/11594244/ee63d666-9a76-11e5-9b77-ffd5f32f18de.png)\r\n\r\nAh - I get your point about the internal index. I'll fix it up and send you a snippet for review ?\r\n\r\nThanks.\r\n\r\n"}}