{"action":"created","issue":{"url":"https://api.github.com/repos/whatwg/streams/issues/301","labels_url":"https://api.github.com/repos/whatwg/streams/issues/301/labels{/name}","comments_url":"https://api.github.com/repos/whatwg/streams/issues/301/comments","events_url":"https://api.github.com/repos/whatwg/streams/issues/301/events","html_url":"https://github.com/whatwg/streams/issues/301","id":62594727,"number":301,"title":"Generalized and precise flow/buffer control","user":{"login":"tyoshino","id":4511440,"avatar_url":"https://avatars.githubusercontent.com/u/4511440?v=3","gravatar_id":"","url":"https://api.github.com/users/tyoshino","html_url":"https://github.com/tyoshino","followers_url":"https://api.github.com/users/tyoshino/followers","following_url":"https://api.github.com/users/tyoshino/following{/other_user}","gists_url":"https://api.github.com/users/tyoshino/gists{/gist_id}","starred_url":"https://api.github.com/users/tyoshino/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tyoshino/subscriptions","organizations_url":"https://api.github.com/users/tyoshino/orgs","repos_url":"https://api.github.com/users/tyoshino/repos","events_url":"https://api.github.com/users/tyoshino/events{/privacy}","received_events_url":"https://api.github.com/users/tyoshino/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/whatwg/streams/labels/queuing%20strategies","name":"queuing strategies","color":"5319e7"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-03-18T05:34:50Z","updated_at":"2015-03-20T06:39:59Z","closed_at":null,"body":"Before recent redesign, we always had a queue in the JS world behind a readable stream, and we calculated how to exert backpressure to the \"underlying source\" based on the size of the queue.\r\n\r\nWe're now separating the streams API semantics visible to the consumer (mainly implemented in the reader) and a queue behind it which doesn't always exist (depending on who (JS, C++ code, kernel, device, etc.) buffers data). This made me think more generally, that is, there're many points of buffering behind streams. The number of such points depends on which API we're using, which platform we're using, etc.\r\n\r\nIt sounds like both the high water mark & strategy concept, and the precise flow control stuff I proposed only make sense for streams with a queue in JS world. It seems it's hard to well-define `queueSize`, `highWaterMark`, `pullAmount`, etc. as cross-platform, cross-API feature.\r\n\r\nTo put it differently, for non-JS-queue-backed APIs, what do we guarantee for the consumer when HWM is set to X?\r\n\r\nI still guess some kind of best-effort approach is still useful. The UA tries to keep the number of bytes buffered on the path between the ultimate data source (disk, the peer server, etc.) and the JS world. As Domenic originally suggested, the strategy itself should be chosen intelligently by the UA considering memory size, CPU power, browser architecture, etc. etc. and it's responsible only for computing how much data is buffered in **the JS world**. Combining buffer size info (C++, kernel, JS, ...), the UA would intelligently limits the amount of data to pull.\r\n\r\nAnd, the `pullAmount` or something in the proposed precise flow control APIs will just give a hint about how quickly the consumer can consume data in the form of the number of bytes. It is not directly connected with such as the size of the queue in JS. The stream utilizes the info using some algorithm to control amount of data to pull."},"comment":{"url":"https://api.github.com/repos/whatwg/streams/issues/comments/83939205","html_url":"https://github.com/whatwg/streams/issues/301#issuecomment-83939205","issue_url":"https://api.github.com/repos/whatwg/streams/issues/301","id":83939205,"user":{"login":"domenic","id":617481,"avatar_url":"https://avatars.githubusercontent.com/u/617481?v=3","gravatar_id":"","url":"https://api.github.com/users/domenic","html_url":"https://github.com/domenic","followers_url":"https://api.github.com/users/domenic/followers","following_url":"https://api.github.com/users/domenic/following{/other_user}","gists_url":"https://api.github.com/users/domenic/gists{/gist_id}","starred_url":"https://api.github.com/users/domenic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/domenic/subscriptions","organizations_url":"https://api.github.com/users/domenic/orgs","repos_url":"https://api.github.com/users/domenic/repos","events_url":"https://api.github.com/users/domenic/events{/privacy}","received_events_url":"https://api.github.com/users/domenic/received_events","type":"User","site_admin":false},"created_at":"2015-03-20T06:39:59Z","updated_at":"2015-03-20T06:39:59Z","body":"My current thinking based on discussions this morning:\r\n\r\n- pull(enqueue, close, error) becomes pull(enqueue, close, error, spaceLeftGetter) in functionality. (In name and shape, maybe `pull({ enqueue(), close(), error(), desiredSize })`? Can be settled afterward though.)\r\n- Strategies need to be modified to have high water mark directly instead of having flexible shouldApplyBackpressure predicate. (Maybe we allow it to be updated, but unsure by who or how.)\r\n- The when-should-we-call-pull logic stays similar to how it is now, with one addition:\r\n  - After start\r\n  - If queue size changes, up or down, and _either_:\r\n    - spaceLeft = HWM - queueSize > 0, or\r\n    - there are pending reads\r\n  - Promise-debounced\r\n\r\nWith this change, and a strategy that says HWM = 0, read() will directly correspond to pull(), which is good. (The \"if there are pending reads\" ensures it.)\r\n\r\nWith a strategy that has HWM > 0, then pull will be called whenever the queue size is less than the high water mark, and it will be called with spaceLeft (or \"desiredSize\" maybe) = HWM - current queue size, which allows it to make intelligent decisions about how to generate chunks that it will enqueue."}}