{"action":"created","issue":{"url":"https://api.github.com/repos/okfn/datapackagist/issues/171","labels_url":"https://api.github.com/repos/okfn/datapackagist/issues/171/labels{/name}","comments_url":"https://api.github.com/repos/okfn/datapackagist/issues/171/comments","events_url":"https://api.github.com/repos/okfn/datapackagist/issues/171/events","html_url":"https://github.com/okfn/datapackagist/issues/171","id":111031938,"number":171,"title":"Move most file handling logic to server","user":{"login":"gvidon","id":58164,"avatar_url":"https://avatars.githubusercontent.com/u/58164?v=3","gravatar_id":"","url":"https://api.github.com/users/gvidon","html_url":"https://github.com/gvidon","followers_url":"https://api.github.com/users/gvidon/followers","following_url":"https://api.github.com/users/gvidon/following{/other_user}","gists_url":"https://api.github.com/users/gvidon/gists{/gist_id}","starred_url":"https://api.github.com/users/gvidon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gvidon/subscriptions","organizations_url":"https://api.github.com/users/gvidon/orgs","repos_url":"https://api.github.com/users/gvidon/repos","events_url":"https://api.github.com/users/gvidon/events{/privacy}","received_events_url":"https://api.github.com/users/gvidon/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/okfn/datapackagist/labels/inwork","name":"inwork","color":"207de5"}],"state":"open","locked":false,"assignee":{"login":"gvidon","id":58164,"avatar_url":"https://avatars.githubusercontent.com/u/58164?v=3","gravatar_id":"","url":"https://api.github.com/users/gvidon","html_url":"https://github.com/gvidon","followers_url":"https://api.github.com/users/gvidon/followers","following_url":"https://api.github.com/users/gvidon/following{/other_user}","gists_url":"https://api.github.com/users/gvidon/gists{/gist_id}","starred_url":"https://api.github.com/users/gvidon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gvidon/subscriptions","organizations_url":"https://api.github.com/users/gvidon/orgs","repos_url":"https://api.github.com/users/gvidon/repos","events_url":"https://api.github.com/users/gvidon/events{/privacy}","received_events_url":"https://api.github.com/users/gvidon/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/okfn/datapackagist/milestones/6","html_url":"https://github.com/okfn/datapackagist/milestones/Current","labels_url":"https://api.github.com/repos/okfn/datapackagist/milestones/6/labels","id":1191481,"number":6,"title":"Current","description":"","creator":{"login":"pwalsh","id":791678,"avatar_url":"https://avatars.githubusercontent.com/u/791678?v=3","gravatar_id":"","url":"https://api.github.com/users/pwalsh","html_url":"https://github.com/pwalsh","followers_url":"https://api.github.com/users/pwalsh/followers","following_url":"https://api.github.com/users/pwalsh/following{/other_user}","gists_url":"https://api.github.com/users/pwalsh/gists{/gist_id}","starred_url":"https://api.github.com/users/pwalsh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pwalsh/subscriptions","organizations_url":"https://api.github.com/users/pwalsh/orgs","repos_url":"https://api.github.com/users/pwalsh/repos","events_url":"https://api.github.com/users/pwalsh/events{/privacy}","received_events_url":"https://api.github.com/users/pwalsh/received_events","type":"User","site_admin":false},"open_issues":7,"closed_issues":37,"state":"open","created_at":"2015-07-02T20:09:14Z","updated_at":"2015-10-15T06:04:04Z","due_on":null,"closed_at":null},"comments":16,"created_at":"2015-10-12T18:41:18Z","updated_at":"2015-10-16T12:48:51Z","closed_at":null,"body":"```tabularfilemanager.fromURL()``` could perfectly replace routine of ```.getDataSource()``` https://github.com/okfn/datapackagist/blob/master/datapackagist/src/scripts/components/ui/resource-editor.js used for fetching and parsing resources files from remote URL.\r\n\r\nIt will also help to avoid hanging on large CSVs, as there is an option defining max part of the CSV to be processed and infered into schema."},"comment":{"url":"https://api.github.com/repos/okfn/datapackagist/issues/comments/148707870","html_url":"https://github.com/okfn/datapackagist/issues/171#issuecomment-148707870","issue_url":"https://api.github.com/repos/okfn/datapackagist/issues/171","id":148707870,"user":{"login":"gvidon","id":58164,"avatar_url":"https://avatars.githubusercontent.com/u/58164?v=3","gravatar_id":"","url":"https://api.github.com/users/gvidon","html_url":"https://github.com/gvidon","followers_url":"https://api.github.com/users/gvidon/followers","following_url":"https://api.github.com/users/gvidon/following{/other_user}","gists_url":"https://api.github.com/users/gvidon/gists{/gist_id}","starred_url":"https://api.github.com/users/gvidon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gvidon/subscriptions","organizations_url":"https://api.github.com/users/gvidon/orgs","repos_url":"https://api.github.com/users/gvidon/repos","events_url":"https://api.github.com/users/gvidon/events{/privacy}","received_events_url":"https://api.github.com/users/gvidon/received_events","type":"User","site_admin":false},"created_at":"2015-10-16T12:48:51Z","updated_at":"2015-10-16T12:48:51Z","body":"Generic upload solution is not suitable for streamed upload or sending file in chunks, as http multipart spec requires file content to be in single request.\r\n\r\nExisting solutions like https://github.com/balderdashy/skipper from what I understand still uploads the whole http request data (request body) to the server and then stream it into different sources, not creating temporary file. It doesn't eliminate upload lag.\r\n\r\nWhat should eliminate upload lag is splitting resource file into chunks on the client and use http request ```Conten-range``` header thttp://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.16 to send data in chunks.\r\n\r\nBut probably much simpler would be to just truncate blob from file input on the client, and send already truncated data to the server\r\n\r\nHere it is how client side truncation is done.\r\n```javascript\r\n// .slice() returns Blob which will be sent to the backend\r\nfile.slice(0, _.min([file.size, config.maxCSVSize]))\r\n```"}}