{"action":"created","issue":{"url":"https://api.github.com/repos/slamdata/slamengine/issues/782","labels_url":"https://api.github.com/repos/slamdata/slamengine/issues/782/labels{/name}","comments_url":"https://api.github.com/repos/slamdata/slamengine/issues/782/comments","events_url":"https://api.github.com/repos/slamdata/slamengine/issues/782/events","html_url":"https://github.com/slamdata/slamengine/issues/782","id":88817814,"number":782,"title":"API: entire result brought into memory when it should be streamed","user":{"login":"mossprescott","id":6147371,"avatar_url":"https://avatars.githubusercontent.com/u/6147371?v=3","gravatar_id":"","url":"https://api.github.com/users/mossprescott","html_url":"https://github.com/mossprescott","followers_url":"https://api.github.com/users/mossprescott/followers","following_url":"https://api.github.com/users/mossprescott/following{/other_user}","gists_url":"https://api.github.com/users/mossprescott/gists{/gist_id}","starred_url":"https://api.github.com/users/mossprescott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mossprescott/subscriptions","organizations_url":"https://api.github.com/users/mossprescott/orgs","repos_url":"https://api.github.com/users/mossprescott/repos","events_url":"https://api.github.com/users/mossprescott/events{/privacy}","received_events_url":"https://api.github.com/users/mossprescott/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/slamdata/slamengine/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/slamdata/slamengine/labels/ready","name":"ready","color":"ededed"}],"state":"open","locked":false,"assignee":{"login":"mossprescott","id":6147371,"avatar_url":"https://avatars.githubusercontent.com/u/6147371?v=3","gravatar_id":"","url":"https://api.github.com/users/mossprescott","html_url":"https://github.com/mossprescott","followers_url":"https://api.github.com/users/mossprescott/followers","following_url":"https://api.github.com/users/mossprescott/following{/other_user}","gists_url":"https://api.github.com/users/mossprescott/gists{/gist_id}","starred_url":"https://api.github.com/users/mossprescott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mossprescott/subscriptions","organizations_url":"https://api.github.com/users/mossprescott/orgs","repos_url":"https://api.github.com/users/mossprescott/repos","events_url":"https://api.github.com/users/mossprescott/events{/privacy}","received_events_url":"https://api.github.com/users/mossprescott/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/slamdata/slamengine/milestones/11","html_url":"https://github.com/slamdata/slamengine/milestones/ASAP","labels_url":"https://api.github.com/repos/slamdata/slamengine/milestones/11/labels","id":1142034,"number":11,"title":"ASAP","description":"","creator":{"login":"jdegoes","id":156745,"avatar_url":"https://avatars.githubusercontent.com/u/156745?v=3","gravatar_id":"","url":"https://api.github.com/users/jdegoes","html_url":"https://github.com/jdegoes","followers_url":"https://api.github.com/users/jdegoes/followers","following_url":"https://api.github.com/users/jdegoes/following{/other_user}","gists_url":"https://api.github.com/users/jdegoes/gists{/gist_id}","starred_url":"https://api.github.com/users/jdegoes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jdegoes/subscriptions","organizations_url":"https://api.github.com/users/jdegoes/orgs","repos_url":"https://api.github.com/users/jdegoes/repos","events_url":"https://api.github.com/users/jdegoes/events{/privacy}","received_events_url":"https://api.github.com/users/jdegoes/received_events","type":"User","site_admin":false},"open_issues":4,"closed_issues":13,"state":"open","created_at":"2015-06-01T01:16:20Z","updated_at":"2015-06-16T19:31:40Z","due_on":null,"closed_at":null},"comments":10,"created_at":"2015-06-16T19:31:00Z","updated_at":"2015-06-17T02:21:41Z","closed_at":null,"body":"This means we would run out of memory and fail to respond if a client tries to stream a very large result (larger than will fit in the heap in the form of `Data` values).\r\n\r\nSee https://github.com/slamdata/slamengine/commit/56ce4ac81a77c16e17a4ffcd5b9723c9c1a7b676#diff-b37ea18e5f5bbd36b3bb04e6f178f4e7R81.\r\n\r\nThe difficulty is how to handle failure. Currently, failures resulting from problems with the request are returned in `Process[PathTask, Data]`, so you have to consume the entire process to know if anything went wrong. "},"comment":{"url":"https://api.github.com/repos/slamdata/slamengine/issues/comments/112624974","html_url":"https://github.com/slamdata/slamengine/issues/782#issuecomment-112624974","issue_url":"https://api.github.com/repos/slamdata/slamengine/issues/782","id":112624974,"user":{"login":"mossprescott","id":6147371,"avatar_url":"https://avatars.githubusercontent.com/u/6147371?v=3","gravatar_id":"","url":"https://api.github.com/users/mossprescott","html_url":"https://github.com/mossprescott","followers_url":"https://api.github.com/users/mossprescott/followers","following_url":"https://api.github.com/users/mossprescott/following{/other_user}","gists_url":"https://api.github.com/users/mossprescott/gists{/gist_id}","starred_url":"https://api.github.com/users/mossprescott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mossprescott/subscriptions","organizations_url":"https://api.github.com/users/mossprescott/orgs","repos_url":"https://api.github.com/users/mossprescott/repos","events_url":"https://api.github.com/users/mossprescott/events{/privacy}","received_events_url":"https://api.github.com/users/mossprescott/received_events","type":"User","site_admin":false},"created_at":"2015-06-17T02:21:41Z","updated_at":"2015-06-17T02:21:41Z","body":"@jdegoes: It's been slow going, and I still only have a partial solution.\r\n\r\nWhat I'm missing now is a way to consume the initial output so I can check for these failures. My initial attempt works only if the failures come back in Process (which we do sometimes, confusingly), not if they come back in PathTask, which seems like place we want them to be.\r\n\r\nI can think of a not-very-smart way to do that (along the lines of `sampleMap` in prettify.scala), and I found a new combinator in Process, `unconstOption`, that seems to offer a cleaner solution, but that was just recently committed and isn't in the scalaz-stream version we're using."}}