{"action":"created","issue":{"url":"https://api.github.com/repos/crate/crate/issues/2565","labels_url":"https://api.github.com/repos/crate/crate/issues/2565/labels{/name}","comments_url":"https://api.github.com/repos/crate/crate/issues/2565/comments","events_url":"https://api.github.com/repos/crate/crate/issues/2565/events","html_url":"https://github.com/crate/crate/issues/2565","id":106058862,"number":2565,"title":"COPY TO exporting partial data","user":{"login":"khalidsalomao","id":1336766,"avatar_url":"https://avatars.githubusercontent.com/u/1336766?v=3","gravatar_id":"","url":"https://api.github.com/users/khalidsalomao","html_url":"https://github.com/khalidsalomao","followers_url":"https://api.github.com/users/khalidsalomao/followers","following_url":"https://api.github.com/users/khalidsalomao/following{/other_user}","gists_url":"https://api.github.com/users/khalidsalomao/gists{/gist_id}","starred_url":"https://api.github.com/users/khalidsalomao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/khalidsalomao/subscriptions","organizations_url":"https://api.github.com/users/khalidsalomao/orgs","repos_url":"https://api.github.com/users/khalidsalomao/repos","events_url":"https://api.github.com/users/khalidsalomao/events{/privacy}","received_events_url":"https://api.github.com/users/khalidsalomao/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-09-11T17:41:59Z","updated_at":"2015-09-13T14:03:35Z","closed_at":null,"body":"`COPY TO` **without** the optional parameter `DIRECTORY` exports data from only one shard. I took some time to figure that out....\r\n\r\nIn my case, the table have 12 shards and the output was to S3, and the generated file had only 1/12 of the total rows.\r\n\r\nIf the parameter `DIRECTORY` is provided, all data is exported to S3, generating 12 files.\r\n\r\nTested with version 0.51.2\r\nOS: Ubuntu 14.04\r\n\r\n\r\n<bountysource-plugin>\r\n---\r\nWant to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/26618956-copy-to-exporting-partial-data?utm_campaign=plugin&utm_content=tracker%2F505659&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F505659&utm_medium=issues&utm_source=github).\r\n</bountysource-plugin>"},"comment":{"url":"https://api.github.com/repos/crate/crate/issues/comments/139879253","html_url":"https://github.com/crate/crate/issues/2565#issuecomment-139879253","issue_url":"https://api.github.com/repos/crate/crate/issues/2565","id":139879253,"user":{"login":"khalidsalomao","id":1336766,"avatar_url":"https://avatars.githubusercontent.com/u/1336766?v=3","gravatar_id":"","url":"https://api.github.com/users/khalidsalomao","html_url":"https://github.com/khalidsalomao","followers_url":"https://api.github.com/users/khalidsalomao/followers","following_url":"https://api.github.com/users/khalidsalomao/following{/other_user}","gists_url":"https://api.github.com/users/khalidsalomao/gists{/gist_id}","starred_url":"https://api.github.com/users/khalidsalomao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/khalidsalomao/subscriptions","organizations_url":"https://api.github.com/users/khalidsalomao/orgs","repos_url":"https://api.github.com/users/khalidsalomao/repos","events_url":"https://api.github.com/users/khalidsalomao/events{/privacy}","received_events_url":"https://api.github.com/users/khalidsalomao/received_events","type":"User","site_admin":false},"created_at":"2015-09-13T14:03:34Z","updated_at":"2015-09-13T14:03:34Z","body":"Thanks @mfelsche, it was my mistake (for some reason I thought that was one file per node instead of per shard...).\r\n\r\nBut, since almost always the index will have multiple shards, this overwrite behavior will cause many mistakes like mine, shouldn't the `DIRECTORY` option be the default behavior?\r\n\r\nAlso, in regards to the parallelism, an interesting feature would be to turn off the parallelism and have only 1 resulting file.\r\n\r\nAWS Redshift has this option for the `UNLOAD` command: [unload](http://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html)\r\n\r\nWhat do you think about these features:\r\n\r\n1. Make `DIRECTORY` option the default.\r\n\r\n2. `COPY TO` with parallelism off to generate only one file (serializing the output of each shard).\r\n\r\n3. New parameter: accept a `SELECT` query as parameter so that the query output is written to file (similar with the Redshift `UNLOAD`)."}}