{"action":"created","issue":{"url":"https://api.github.com/repos/dionyziz/stream.js/issues/25","labels_url":"https://api.github.com/repos/dionyziz/stream.js/issues/25/labels{/name}","comments_url":"https://api.github.com/repos/dionyziz/stream.js/issues/25/comments","events_url":"https://api.github.com/repos/dionyziz/stream.js/issues/25/events","html_url":"https://github.com/dionyziz/stream.js/issues/25","id":119114478,"number":25,"title":"A cry for better documentation ;)","user":{"login":"dotnetCarpenter","id":43763,"avatar_url":"https://avatars.githubusercontent.com/u/43763?v=3","gravatar_id":"","url":"https://api.github.com/users/dotnetCarpenter","html_url":"https://github.com/dotnetCarpenter","followers_url":"https://api.github.com/users/dotnetCarpenter/followers","following_url":"https://api.github.com/users/dotnetCarpenter/following{/other_user}","gists_url":"https://api.github.com/users/dotnetCarpenter/gists{/gist_id}","starred_url":"https://api.github.com/users/dotnetCarpenter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dotnetCarpenter/subscriptions","organizations_url":"https://api.github.com/users/dotnetCarpenter/orgs","repos_url":"https://api.github.com/users/dotnetCarpenter/repos","events_url":"https://api.github.com/users/dotnetCarpenter/events{/privacy}","received_events_url":"https://api.github.com/users/dotnetCarpenter/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-11-27T00:43:57Z","updated_at":"2015-12-04T11:36:59Z","closed_at":null,"body":"After getting this package up to date, I had some more time playing around\r\nwith `stream.js` and I find it quite hard to use. I will give two examples.\r\n\r\nI'm building a parser and it has a lexer which reads text and convert it to tokens.\r\nThe main work of the lexer is a recursive function. It didn't work with large\r\npieces of text as I ran into V8's too much recursion issue (exhausting the call stack).\r\nSo I switch from using an array that was passed on recursively as an accumulator to use `stream.js`.\r\n\r\n```js\r\nfunction lexer(str, options = { newline:false }) {\r\n\tif(str.length === 0) return new Stream\r\n\tif(regexWord.test(str)) {\r\n\t\tlet word = regexWord.exec(str)[0],\r\n\t\t\trest = str.substr(word.length)\r\n\t\treturn new Stream(new Word(word), () => read(rest, options))\r\n\t}\r\n\tif(regexNewLine.test(str)) {\r\n\t\tlet newline = regexNewLine.exec(str)[0],\r\n\t\t\trest = str.substr(newline.length)\r\n\t\tif(options.newline)\r\n\t\t\treturn new Stream(new NewLine(newline), () => read(rest, options))\r\n\t\treturn read(rest, options)\r\n\t}\r\n}\r\n```\r\n*Word and NewLine are token data types and Stream is `stream.js`*\r\n\r\nAfter the tokenization I needed to get access to some of the tokens, enought to do something\r\nmeaningful and not too many to course a recursion issue. First I tried with `take`.\r\n\r\nWhile that gave me some tokens, it didn't change the length of the stream. Imagine the\r\nfollowing:\r\n\r\n```js\r\nlet stream = lexer(str, options)\r\nlet scheduleId = setInterval(() => {\r\n\tlet tokens = stream.take(64)\r\n\tparse(tokens)\r\n\r\n\tif(stream.empty())\r\n\t\tclearInterval(scheduleId)\r\n}, 0)\r\n```\r\n\r\nThe parse function will get the same 64 tokens all the time. I then tried with `drop`.\r\nThis change the length of the `stream` but the parse function was still getting the same\r\ntokens, over and over. Since my word list has pretty similar words, it took me a while\r\nto figure out what was going on. The two functions `take` and `drop` need to be combined.\r\n\r\n```js\r\nlet tokens = stream.take(64)\r\nstream = stream.drop(64)\r\n```\r\n\r\nFortunately, `stream.js` doesn't error when taking or dropping more from a stream than it\r\nhas, making the solution pretty easy.\r\nHowever, since the only documentation `stream.js` has is the [website][2] and\r\nsource code, it took way longer than necessary to figure out.\r\n\r\nLet's take [the code that sends JSON data to codecov][1] as the second example.\r\n[sendtocodecov.js][1] reads the coverage.json file that is generated with `npm test`.\r\nIt then starts the codecov code in a child process and pipe the JSON data to its stdin\r\nstream. It does that because the codecov code will throw an error if run locally,\r\ncomplaining that it can't find the CODECOV_TOKEN environment variable. Resulting in a not\r\nso pleasent error whenever we run the test suit with coverage.\r\n\r\n```\r\nLines        : 83.93% ( 141/168 )\r\n================================================================================\r\nC:\\Users\\dotnet\\Projects\\opensource\\stream.js\\node_modules\\codecov.io\\lib\\getConfiguration.js:27\r\n      throw new Error(\"unknown service. could not get configuration\");\r\n      ^\r\n\r\nError: unknown service. could not get configuration\r\n    at module.exports (C:\\Users\\dotnet\\Projects\\opensource\\stream.js\\node_modules\\codecov.io\\lib\\getConfiguration.js:27:13)\r\n    at sendToCodecov (C:\\Users\\dotnet\\Projects\\opensource\\stream.js\\node_modules\\codecov.io\\lib\\sendToCodeCov.io.js:11:15)\r\n    at Socket.<anonymous> (C:\\Users\\dotnet\\Projects\\opensource\\stream.js\\node_modules\\codecov.io\\bin\\codecov.io.js:14:5)\r\n    at emitNone (events.js:72:20)\r\n    at Socket.emit (events.js:166:7)\r\n    at endReadableNT (_stream_readable.js:905:12)\r\n    at doNTCallback2 (node.js:452:9)\r\n    at process._tickCallback (node.js:366:17)\r\nnpm ERR! Test failed.  See above for more details.\r\n```\r\n\r\nIt works but it is written in an imperative style and since we can read the JSON\r\ndata as a stream, why not utilize `stream.js`?\r\n\r\nWe need to change the `fs.readFile` to create a stream and then we need to build our\r\n`stream.js` stream from it.\r\n\r\nBy looking at the API we can try and guess what method will add to a stream. `add` is\r\nthere so let's try with that.\r\n\r\n```js\r\nvar coverageDataStream = fs.createReadStream(coveragePath, { encoding: 'utf8' });\r\nvar stream = new Stream(null);\r\ncoverageDataStream.on('data', function(data) {\r\n  stream.add(Stream.make(data));\r\n});\r\ncoverageDataStream.on('end', function() {\r\n  stream.print();\r\n});\r\ncoverageDataStream.read();\r\n```\r\n\r\nThe above will print nothing but `null`. We have to know that `stream.js` work with immutable\r\ndata, so any method wont mutate the stream but return a new stream object. If you have a FP\r\nbackground that might be obvious, if not then you're left scratching your head. Knowing this,\r\nwe set the return stream of `add` to our stream variable.\r\n\r\n`stream = stream.add(Stream.make(data));` \r\n\r\nThis looks good. We're getting our data. But there is a `null` value right at the top. Let's\r\ncreate an empty stream and add to that instead of our `null` stream. Turns out we can't.\r\n\r\n```\r\nC:\\Users\\dotnet\\Projects\\opensource\\stream.js\\lib\\index.js:35\r\n            throw new Error('Cannot use item() on an empty stream.');\r\n```\r\n\r\nOk, but we can just take the value of interest, namely the second value. Instead of `print`\r\nwe use `console.log` to print out our desired data. `console.log(stream.item(1))` But this gives\r\nus an error, `Error: Item index does not exist in stream.` It is now clear that `add` isn't\r\nthe right method to use. `add` will add a value to the `head` of our stream. But there is\r\nalso `append`. Let's see.\r\n\r\n```js\r\nvar coverageDataStream = fs.createReadStream(coveragePath, { encoding: 'utf8' });\r\nvar stream = new Stream(null);\r\ncoverageDataStream.on('data', function(data) {\r\n  stream = stream.append(Stream.make(data));\r\n});\r\ncoverageDataStream.on('end', function() {\r\n  console.log(stream.item(1))\r\n});\r\ncoverageDataStream.read();\r\n```\r\n\r\nThis of course only works if we read all the JSON data in one go. So let's quickly change\r\n`item` to `drop(1)` and pass the stream to `pipeToCodeCov`.\r\n\r\nA final solution could look like below. It still isn't FP as we have no pure functions but\r\nit does use `stream.js`.\r\n\r\n```js\r\n#!/usr/bin/env node\r\n\r\n'use strict';\r\n\r\nvar Stream = require('../lib/index');\r\nvar fs = require('fs');\r\nvar spawn = require('child_process').spawn;\r\nvar path = require('path');\r\nvar coveragePath = path.resolve(__dirname, '..', 'coverage/coverage.json');\r\nvar coverageDataStream = fs.createReadStream(coveragePath, { encoding: 'utf8' });\r\nvar stream = new Stream(null);\r\n\r\ncoverageDataStream.on('data', function(data) {\r\n  stream = stream.append(Stream.make(data));\r\n});\r\ncoverageDataStream.on('end', function() {\r\n  stream = stream.drop(1);\r\n  pipeToCodeCov(stream);\r\n});\r\ncoverageDataStream.read();\r\n\r\nfunction pipeToCodeCov(jsonStream) {\r\n  var codecov = 'node_modules/.bin/codecov';\r\n  var child;\r\n  var node = process.execPath;\r\n  var codecov = path.resolve(__dirname, '..', codecov);\r\n  child = spawn(node, [codecov], { stdio: 'pipe' });\r\n  child.on('error', function(data) {\r\n    console.error('codecov lib is not accessible');\r\n  });\r\n  child.stdout.on('data', function(data) {\r\n    console.log(data.toString());\r\n  });\r\n  child.stdin.setEncoding = 'utf8';\r\n  jsonStream.walk(function(json) {\r\n    child.stdin.write(json);\r\n  })\r\n  child.stdin.end();\r\n}\r\n```\r\n\r\nMy main point is that `stream.js` seriously lacks documentation. I also feel that the\r\n[website][2] could explain a little about how `stream.js` deals with immutable data.\r\nDebugging `stream.js` to get answers, is like running through a maze. Don't get me wrong,\r\nI really like `stream.js` and how its implementation doesn't rely on arrays. I'm just saying\r\nthat the learning curve is rather steep when you don't have a FP background, even though\r\nthe [website][2] is enjoyable to read but unfortunately not very comprehensive.\r\n\r\n[1]: https://github.com/dionyziz/stream.js/blob/master/external/sendtocodecov.js\r\n[2]: http://streamjs.org/\r\n"},"comment":{"url":"https://api.github.com/repos/dionyziz/stream.js/issues/comments/161946441","html_url":"https://github.com/dionyziz/stream.js/issues/25#issuecomment-161946441","issue_url":"https://api.github.com/repos/dionyziz/stream.js/issues/25","id":161946441,"user":{"login":"dionyziz","id":544572,"avatar_url":"https://avatars.githubusercontent.com/u/544572?v=3","gravatar_id":"","url":"https://api.github.com/users/dionyziz","html_url":"https://github.com/dionyziz","followers_url":"https://api.github.com/users/dionyziz/followers","following_url":"https://api.github.com/users/dionyziz/following{/other_user}","gists_url":"https://api.github.com/users/dionyziz/gists{/gist_id}","starred_url":"https://api.github.com/users/dionyziz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dionyziz/subscriptions","organizations_url":"https://api.github.com/users/dionyziz/orgs","repos_url":"https://api.github.com/users/dionyziz/repos","events_url":"https://api.github.com/users/dionyziz/events{/privacy}","received_events_url":"https://api.github.com/users/dionyziz/received_events","type":"User","site_admin":false},"created_at":"2015-12-04T11:36:59Z","updated_at":"2015-12-04T11:36:59Z","body":"OK, I'll do a sketch for the API.\r\n\r\nstream.js-website sounds good, let's do that. I've [created a repo](https://github.com/dionyziz/stream.js-website) for this where I'll upload the files soon.\r\n\r\nI host it on my own server, but gh-pages sounds like a good option – let's plan to migrate there."}}