{"action":"created","issue":{"url":"https://api.github.com/repos/zeke/subtlex-word-frequencies/issues/1","labels_url":"https://api.github.com/repos/zeke/subtlex-word-frequencies/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/zeke/subtlex-word-frequencies/issues/1/comments","events_url":"https://api.github.com/repos/zeke/subtlex-word-frequencies/issues/1/events","html_url":"https://github.com/zeke/subtlex-word-frequencies/issues/1","id":97866024,"number":1,"title":"How can I keep this script from crashing?","user":{"login":"zeke","id":2289,"avatar_url":"https://avatars.githubusercontent.com/u/2289?v=3","gravatar_id":"","url":"https://api.github.com/users/zeke","html_url":"https://github.com/zeke","followers_url":"https://api.github.com/users/zeke/followers","following_url":"https://api.github.com/users/zeke/following{/other_user}","gists_url":"https://api.github.com/users/zeke/gists{/gist_id}","starred_url":"https://api.github.com/users/zeke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zeke/subscriptions","organizations_url":"https://api.github.com/users/zeke/orgs","repos_url":"https://api.github.com/users/zeke/repos","events_url":"https://api.github.com/users/zeke/events{/privacy}","received_events_url":"https://api.github.com/users/zeke/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/zeke/subtlex-word-frequencies/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-07-29T05:52:00Z","updated_at":"2015-07-29T15:02:44Z","closed_at":null,"body":"The [index.js](https://github.com/zeke/subtlex-word-frequencies/blob/master/index.js) script in this repo reads from a text file and writes a JSON object to STDOUT with a count of each word in the file:\r\n\r\n```json\r\n\r\n[\r\n  {\r\n    \"word\": \"you\",\r\n    \"count\": 11\r\n  },{\r\n    \"word\": \"to\",\r\n    \"count\": 10\r\n  },{\r\n    \"word\": \"i\",\r\n    \"count\": 9\r\n  }\r\n]\r\n```\r\n\r\n When I use a small sample file, the program runs quickly and I get the expected output.  When I use a **250 MB** text file, however, the program crashes:\r\n\r\n```\r\n❯ npm run build\r\n\r\n> subtlex-word-frequencies@1.0.0 build /Users/z/personal/subtlex-word-frequencies\r\n> node index.js\r\n\r\nFATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory\r\n```\r\n\r\nHow can I refactor this code to be more memory efficient and/or keep the process from crashing?\r\n\r\n```js\r\nconst read = require('fs').readFileSync\r\nconst sortBy = require('lodash').sortBy\r\n\r\nvar counts = {}\r\nvar words = []\r\n\r\n// Tally up instances of each word in the corpus\r\nread('./data/sample.txt', 'utf8')\r\n  .toLowerCase()\r\n  .match(/[\\w']+/g)\r\n  .forEach(function (word) {\r\n    counts[word] = counts[word] ? counts[word] + 1 : 1\r\n  })\r\n\r\n// Create an array of word objects tha can be easily sorted by `count`\r\nObject.keys(counts).forEach(function (word) {\r\n  words.push({\r\n    word: word,\r\n    count: counts[word]\r\n  })\r\n})\r\n\r\nwords = sortBy(words, 'count').reverse()\r\n\r\nprocess.stdout.write(JSON.stringify(words, null, 2))\r\n```\r\n\r\nI'm running iojs 2.2.1 on 2015 Macbook Pro with 16 GB RAM.\r\n\r\nThe big corpus file is at http://subtlexus.lexique.org/corpus/Subtlex%20US.rar --- 0_o"},"comment":{"url":"https://api.github.com/repos/zeke/subtlex-word-frequencies/issues/comments/125981365","html_url":"https://github.com/zeke/subtlex-word-frequencies/issues/1#issuecomment-125981365","issue_url":"https://api.github.com/repos/zeke/subtlex-word-frequencies/issues/1","id":125981365,"user":{"login":"zeke","id":2289,"avatar_url":"https://avatars.githubusercontent.com/u/2289?v=3","gravatar_id":"","url":"https://api.github.com/users/zeke","html_url":"https://github.com/zeke","followers_url":"https://api.github.com/users/zeke/followers","following_url":"https://api.github.com/users/zeke/following{/other_user}","gists_url":"https://api.github.com/users/zeke/gists{/gist_id}","starred_url":"https://api.github.com/users/zeke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zeke/subscriptions","organizations_url":"https://api.github.com/users/zeke/orgs","repos_url":"https://api.github.com/users/zeke/repos","events_url":"https://api.github.com/users/zeke/events{/privacy}","received_events_url":"https://api.github.com/users/zeke/received_events","type":"User","site_admin":false},"created_at":"2015-07-29T15:02:44Z","updated_at":"2015-07-29T15:02:44Z","body":"What a nice thing to wake up to. :)"}}