{"action":"created","comment":{"url":"https://api.github.com/repos/atom/atom/pulls/comments/30718717","id":30718717,"diff_hunk":"@@ -0,0 +1,85 @@\n+{SoftTab, HardTab, PairedCharacter, SoftWrapIndent} = require './special-token-symbols'\n+\n+module.exports =\n+class TokenIterator\n+  constructor: (line) ->\n+    @reset(line) if line?\n+\n+  reset: (@line) ->\n+    @index = null\n+    @bufferStart = @line.startBufferColumn\n+    @bufferEnd = @bufferStart\n+    @screenStart = 0\n+    @screenEnd = 0\n+    @scopes = @line.openScopes.map (id) -> atom.grammars.scopeForId(id)\n+    @scopeStarts = @scopes.slice()\n+    @scopeEnds = []\n+    this\n+\n+  next: ->\n+    {tags} = @line\n+\n+    if @index?\n+      @index++\n+      @scopeEnds.length = 0\n+      @scopeStarts.length = 0\n+      @bufferStart = @bufferEnd\n+      @screenStart = @screenEnd\n+    else\n+      @index = 0\n+\n+    while @index < tags.length\n+      tag = tags[@index]\n+      if tag < 0\n+        if tag % 2 is 0\n+          @scopeEnds.push(atom.grammars.scopeForId(tag + 1))\n+          @scopes.pop()\n+        else\n+          scope = atom.grammars.scopeForId(tag)\n+          @scopeStarts.push(scope)\n+          @scopes.push(scope)\n+        @index++\n+      else\n+        if @isHardTab()\n+          @screenEnd = @screenStart + tag\n+          @bufferEnd = @bufferStart + 1\n+        else if @isSoftWrapIndentation()\n+          @screenEnd = @screenStart + tag\n+          @bufferEnd = @bufferStart + 0\n+        else\n+          @screenEnd = @screenStart + tag\n+          @bufferEnd = @bufferStart + tag\n+        return true\n+\n+    false\n+\n+  getBufferStart: -> @bufferStart\n+  getBufferEnd: -> @bufferEnd\n+\n+  getScreenStart: -> @screenStart\n+  getScreenEnd: -> @screenEnd\n+\n+  getScopeStarts: -> @scopeStarts\n+  getScopeEnds: -> @scopeEnds\n+\n+  getScopes: -> @scopes\n+\n+  getText: ->\n+    @line.text.substring(@screenStart, @screenEnd)\n+\n+  isSoftTab: ->\n+    @line.specialTokens[@index] is SoftTab\n+\n+  isHardTab: ->\n+    @line.specialTokens[@index] is HardTab\n+\n+  isSoftWrapIndentation: ->\n+    @line.specialTokens[@index] is SoftWrapIndent\n+\n+  isPairedCharacter: ->\n+    @line.specialTokens[@index] is PairedCharacter\n+\n+  isAtomic: ->\n+    @isSoftTab() or @isHardTab() or @isSoftWrapIndentation() or @isPairedCharacter()\n+\n+TokenIterator.instance = new TokenIterator","path":"src/token-iterator.coffee","position":85,"original_position":85,"commit_id":"a109b3811c61f6115a398a757f09feae648f3eb7","original_commit_id":"a109b3811c61f6115a398a757f09feae648f3eb7","user":{"login":"maxbrunsfeld","id":326587,"avatar_url":"https://avatars.githubusercontent.com/u/326587?v=3","gravatar_id":"","url":"https://api.github.com/users/maxbrunsfeld","html_url":"https://github.com/maxbrunsfeld","followers_url":"https://api.github.com/users/maxbrunsfeld/followers","following_url":"https://api.github.com/users/maxbrunsfeld/following{/other_user}","gists_url":"https://api.github.com/users/maxbrunsfeld/gists{/gist_id}","starred_url":"https://api.github.com/users/maxbrunsfeld/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maxbrunsfeld/subscriptions","organizations_url":"https://api.github.com/users/maxbrunsfeld/orgs","repos_url":"https://api.github.com/users/maxbrunsfeld/repos","events_url":"https://api.github.com/users/maxbrunsfeld/events{/privacy}","received_events_url":"https://api.github.com/users/maxbrunsfeld/received_events","type":"User","site_admin":true},"body":"> I was trying to avoid adding yet another reference that had to be stored on TokenizedLine.\r\n\r\nYeah, I agree that that's very worthwhile. I was just wondering if it could be cached at a higher level. `TokenizedLine` only uses the `TokenIterator` if a package calls `::getTokenIterator`, right? In that case, could we just instantiate a fresh one?","created_at":"2015-05-20T16:05:14Z","updated_at":"2015-05-20T16:05:14Z","html_url":"https://github.com/atom/atom/pull/6757#discussion_r30718717","pull_request_url":"https://api.github.com/repos/atom/atom/pulls/6757","_links":{"self":{"href":"https://api.github.com/repos/atom/atom/pulls/comments/30718717"},"html":{"href":"https://github.com/atom/atom/pull/6757#discussion_r30718717"},"pull_request":{"href":"https://api.github.com/repos/atom/atom/pulls/6757"}}},"pull_request":{"url":"https://api.github.com/repos/atom/atom/pulls/6757","id":35261939,"html_url":"https://github.com/atom/atom/pull/6757","diff_url":"https://github.com/atom/atom/pull/6757.diff","patch_url":"https://github.com/atom/atom/pull/6757.patch","issue_url":"https://api.github.com/repos/atom/atom/issues/6757","number":6757,"state":"open","locked":false,"title":"Use 70% less memory for TokenizedLines by using a different representation","user":{"login":"nathansobo","id":1789,"avatar_url":"https://avatars.githubusercontent.com/u/1789?v=3","gravatar_id":"","url":"https://api.github.com/users/nathansobo","html_url":"https://github.com/nathansobo","followers_url":"https://api.github.com/users/nathansobo/followers","following_url":"https://api.github.com/users/nathansobo/following{/other_user}","gists_url":"https://api.github.com/users/nathansobo/gists{/gist_id}","starred_url":"https://api.github.com/users/nathansobo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nathansobo/subscriptions","organizations_url":"https://api.github.com/users/nathansobo/orgs","repos_url":"https://api.github.com/users/nathansobo/repos","events_url":"https://api.github.com/users/nathansobo/events{/privacy}","received_events_url":"https://api.github.com/users/nathansobo/received_events","type":"User","site_admin":true},"body":":rotating_light: Work In Progress :rotating_light:\r\n\r\n**Note: This currently depends on linking in [a branch of first-mate](https://github.com/atom/first-mate/pull/45) with `npm link`.**\r\n\r\nPreviously, we naively stored every token on every line as its own object, and made things even worse by storing an array of scopes on every token. This caused `TokenizedLine`s to consume much more memory than is strictly necessary.\r\n\r\nThis PR switches `TokenizedLine` to a much more compact (but also more cryptic) representation. Instead of repeating the scopes on each token, we instead represent the entire tokenized line as a *nested word*, which intermingles start and end tags with text much like HTML.\r\n\r\nConsider the following JavaScript fragment:\r\n\r\n```js\r\nfunction() {}\r\n```\r\n\r\nPreviously, we would store this as an array of tokens, each with an array of its containing scopes:\r\n\r\n```js\r\n[\r\n  {value: 'function', scopes: ['source.js', 'meta.function.js', 'storage.type.function.js']},\r\n  {value: ' ', scopes: ['source.js', 'meta.function.js']},\r\n  {value: '(', scopes: ['source.js', 'meta.function.js', 'punctuation.definition.parameters.begin.js']},\r\n  {value: ')', scopes: ['source.js', 'meta.function.js', 'punctuation.definition.parameters.end.js']},\r\n  {value: ' ', scopes: ['source.js', 'meta.function.js']},\r\n  {value: '{', scopes: ['source.js', 'punctuation.section.scope.begin.js']},\r\n  {value: '}', scopes: ['source.js', 'punctuation.section.scope.end.js']}\r\n]\r\n```\r\n\r\nNow this is stored more like HTML, with strings intermingled with scope start / end tags.\r\n\r\n```js\r\n// Note: This JSON is just for example purposes. Read below for the actual representation.\r\n\r\n[\r\n  {start: 'source.js'},\r\n    {start: 'meta.function.js'},\r\n      {start: 'storage.type.function.js'},\r\n        \"function\",\r\n      {end: 'storage.type.function.js'},\r\n      \" \",\r\n      {start: 'punctuation.definition.parameters.begin.js'},\r\n        \"(\",\r\n      {end: 'punctuation.definition.parameters.begin.js'},\r\n      {start: 'punctuation.definition.parameters.end.js'},\r\n        \")\",\r\n      {end: 'punctuation.definition.parameters.end.js'},\r\n    {end: 'meta.function.js'},\r\n    \" \",\r\n    {start: 'punctuation.section.scope.begin.js'},\r\n      \"{\",\r\n    {end: 'punctuation.section.scope.begin.js'},\r\n    {start: 'punctuation.section.scope.end.js'},\r\n      \"}\",\r\n    {end: 'punctuation.section.scope.end.js'},\r\n  // note 'source.js' isn't closed yet\r\n]\r\n```\r\n\r\nThis may look more verbose, but it eliminates the need to repeat the same scopes on successive tokens in the same branch of the syntax tree. In actuality, the scopes are encoded as simple arrays of integers. Negative integers indicate start and end tags, with odd integers being start tags and even integers being end tags for the odd integer preceding them. Positive integers stand in for strings. The integers can be translated back to scope strings by consulting a lookup table on the grammar registry. We retain the original string and combine the two pieces of data as needed.\r\n\r\nSo in reality, the stored representation looks like this.\r\n\r\n```js\r\n{\r\n  text: \"function() {}\",\r\n  tags: [-1, -3, -5, 8, -6, 1, -7, 1, -8, -9, 1, -10, -4, 1, -11, 1, -12, -13, 1, -14]\r\n}\r\n```\r\n\r\n### Measurements\r\n\r\nHere are some heap snapshots taken with `benchmark/fixtures/huge.js` in the Atom repository. This file is just a copy of jQuery 1.7.1, but has some good examples of lines with extremely complex scope structures.\r\n\r\nFirst, on master:\r\n\r\n![screenshot 2015-05-14 19 07 26](https://cloud.githubusercontent.com/assets/1789/7637260/e40779b6-fa6c-11e4-8441-ef42af5e8f9f.png)\r\n\r\n`TokenizedLine` instances consume over 27MB. The largest instance is expanded in the screenshot, where you can see the vast majority of the memory for that line being consumed by the `tokens` array (~25KB). Repeat that for thousands of lines and you can see the problem.\r\n\r\nNow, the same file on this branch:\r\n\r\n![screenshot 2015-05-14 19 09 57](https://cloud.githubusercontent.com/assets/1789/7637303/40a52bdc-fa6d-11e4-88b3-e64479cccda3.png)\r\n\r\nStill more memory than I'd like to consume, but much less, around 8.4MB. You can see that the `tags` array takes up much less space than the `tokens` array, but still more than it seems like it should. I can save more with typed arrays, but only about 500KB in my experiment.\r\n\r\nJS objects still seem to impose a frustrating amount of overhead. Starting to wonder if we want to drop some of our low-level storage structures to C++, but I'd rather improve the structure of this layer before contemplating that. The `TokenizedLine` instances are also now much simpler objects, meaning they terminate in primitives much more quickly. I wonder if this helps the GC in that the heap is simpler and there are fewer objects to trace. This is just speculation however.\r\n\r\n### Steps\r\n\r\n- [x] Get core specs passing with new representation\r\n- [x] Eliminate usages of the `tokens` array shim in core, which creates a lot of garbage\r\n- [x] Post some heap snapshot comparisons\r\n- [x] Publish a new major revision of first-mate and link it in package.json\r\n- [x] Ensure all package specs pass\r\n- [ ] Switch hot package paths off the tokens shim","created_at":"2015-05-12T19:32:03Z","updated_at":"2015-05-20T16:05:14Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bab5bf31cecd213a58e80d46a79c1e3a686a3d67","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/atom/atom/pulls/6757/commits","review_comments_url":"https://api.github.com/repos/atom/atom/pulls/6757/comments","review_comment_url":"https://api.github.com/repos/atom/atom/pulls/comments{/number}","comments_url":"https://api.github.com/repos/atom/atom/issues/6757/comments","statuses_url":"https://api.github.com/repos/atom/atom/statuses/a109b3811c61f6115a398a757f09feae648f3eb7","head":{"label":"atom:ns-less-memory-for-tokens","ref":"ns-less-memory-for-tokens","sha":"a109b3811c61f6115a398a757f09feae648f3eb7","user":{"login":"atom","id":1089146,"avatar_url":"https://avatars.githubusercontent.com/u/1089146?v=3","gravatar_id":"","url":"https://api.github.com/users/atom","html_url":"https://github.com/atom","followers_url":"https://api.github.com/users/atom/followers","following_url":"https://api.github.com/users/atom/following{/other_user}","gists_url":"https://api.github.com/users/atom/gists{/gist_id}","starred_url":"https://api.github.com/users/atom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/atom/subscriptions","organizations_url":"https://api.github.com/users/atom/orgs","repos_url":"https://api.github.com/users/atom/repos","events_url":"https://api.github.com/users/atom/events{/privacy}","received_events_url":"https://api.github.com/users/atom/received_events","type":"Organization","site_admin":false},"repo":{"id":3228505,"name":"atom","full_name":"atom/atom","owner":{"login":"atom","id":1089146,"avatar_url":"https://avatars.githubusercontent.com/u/1089146?v=3","gravatar_id":"","url":"https://api.github.com/users/atom","html_url":"https://github.com/atom","followers_url":"https://api.github.com/users/atom/followers","following_url":"https://api.github.com/users/atom/following{/other_user}","gists_url":"https://api.github.com/users/atom/gists{/gist_id}","starred_url":"https://api.github.com/users/atom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/atom/subscriptions","organizations_url":"https://api.github.com/users/atom/orgs","repos_url":"https://api.github.com/users/atom/repos","events_url":"https://api.github.com/users/atom/events{/privacy}","received_events_url":"https://api.github.com/users/atom/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/atom/atom","description":"The hackable editor","fork":false,"url":"https://api.github.com/repos/atom/atom","forks_url":"https://api.github.com/repos/atom/atom/forks","keys_url":"https://api.github.com/repos/atom/atom/keys{/key_id}","collaborators_url":"https://api.github.com/repos/atom/atom/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/atom/atom/teams","hooks_url":"https://api.github.com/repos/atom/atom/hooks","issue_events_url":"https://api.github.com/repos/atom/atom/issues/events{/number}","events_url":"https://api.github.com/repos/atom/atom/events","assignees_url":"https://api.github.com/repos/atom/atom/assignees{/user}","branches_url":"https://api.github.com/repos/atom/atom/branches{/branch}","tags_url":"https://api.github.com/repos/atom/atom/tags","blobs_url":"https://api.github.com/repos/atom/atom/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/atom/atom/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/atom/atom/git/refs{/sha}","trees_url":"https://api.github.com/repos/atom/atom/git/trees{/sha}","statuses_url":"https://api.github.com/repos/atom/atom/statuses/{sha}","languages_url":"https://api.github.com/repos/atom/atom/languages","stargazers_url":"https://api.github.com/repos/atom/atom/stargazers","contributors_url":"https://api.github.com/repos/atom/atom/contributors","subscribers_url":"https://api.github.com/repos/atom/atom/subscribers","subscription_url":"https://api.github.com/repos/atom/atom/subscription","commits_url":"https://api.github.com/repos/atom/atom/commits{/sha}","git_commits_url":"https://api.github.com/repos/atom/atom/git/commits{/sha}","comments_url":"https://api.github.com/repos/atom/atom/comments{/number}","issue_comment_url":"https://api.github.com/repos/atom/atom/issues/comments{/number}","contents_url":"https://api.github.com/repos/atom/atom/contents/{+path}","compare_url":"https://api.github.com/repos/atom/atom/compare/{base}...{head}","merges_url":"https://api.github.com/repos/atom/atom/merges","archive_url":"https://api.github.com/repos/atom/atom/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/atom/atom/downloads","issues_url":"https://api.github.com/repos/atom/atom/issues{/number}","pulls_url":"https://api.github.com/repos/atom/atom/pulls{/number}","milestones_url":"https://api.github.com/repos/atom/atom/milestones{/number}","notifications_url":"https://api.github.com/repos/atom/atom/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/atom/atom/labels{/name}","releases_url":"https://api.github.com/repos/atom/atom/releases{/id}","created_at":"2012-01-20T18:18:21Z","updated_at":"2015-05-20T15:52:29Z","pushed_at":"2015-05-20T15:58:01Z","git_url":"git://github.com/atom/atom.git","ssh_url":"git@github.com:atom/atom.git","clone_url":"https://github.com/atom/atom.git","svn_url":"https://github.com/atom/atom","homepage":"https://atom.io","size":363388,"stargazers_count":15478,"watchers_count":15478,"language":"CoffeeScript","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2719,"mirror_url":null,"open_issues_count":1087,"forks":2719,"open_issues":1087,"watchers":15478,"default_branch":"master"}},"base":{"label":"atom:master","ref":"master","sha":"290b28c5b90b074c68e2e431aaa03b64b7122e93","user":{"login":"atom","id":1089146,"avatar_url":"https://avatars.githubusercontent.com/u/1089146?v=3","gravatar_id":"","url":"https://api.github.com/users/atom","html_url":"https://github.com/atom","followers_url":"https://api.github.com/users/atom/followers","following_url":"https://api.github.com/users/atom/following{/other_user}","gists_url":"https://api.github.com/users/atom/gists{/gist_id}","starred_url":"https://api.github.com/users/atom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/atom/subscriptions","organizations_url":"https://api.github.com/users/atom/orgs","repos_url":"https://api.github.com/users/atom/repos","events_url":"https://api.github.com/users/atom/events{/privacy}","received_events_url":"https://api.github.com/users/atom/received_events","type":"Organization","site_admin":false},"repo":{"id":3228505,"name":"atom","full_name":"atom/atom","owner":{"login":"atom","id":1089146,"avatar_url":"https://avatars.githubusercontent.com/u/1089146?v=3","gravatar_id":"","url":"https://api.github.com/users/atom","html_url":"https://github.com/atom","followers_url":"https://api.github.com/users/atom/followers","following_url":"https://api.github.com/users/atom/following{/other_user}","gists_url":"https://api.github.com/users/atom/gists{/gist_id}","starred_url":"https://api.github.com/users/atom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/atom/subscriptions","organizations_url":"https://api.github.com/users/atom/orgs","repos_url":"https://api.github.com/users/atom/repos","events_url":"https://api.github.com/users/atom/events{/privacy}","received_events_url":"https://api.github.com/users/atom/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/atom/atom","description":"The hackable editor","fork":false,"url":"https://api.github.com/repos/atom/atom","forks_url":"https://api.github.com/repos/atom/atom/forks","keys_url":"https://api.github.com/repos/atom/atom/keys{/key_id}","collaborators_url":"https://api.github.com/repos/atom/atom/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/atom/atom/teams","hooks_url":"https://api.github.com/repos/atom/atom/hooks","issue_events_url":"https://api.github.com/repos/atom/atom/issues/events{/number}","events_url":"https://api.github.com/repos/atom/atom/events","assignees_url":"https://api.github.com/repos/atom/atom/assignees{/user}","branches_url":"https://api.github.com/repos/atom/atom/branches{/branch}","tags_url":"https://api.github.com/repos/atom/atom/tags","blobs_url":"https://api.github.com/repos/atom/atom/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/atom/atom/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/atom/atom/git/refs{/sha}","trees_url":"https://api.github.com/repos/atom/atom/git/trees{/sha}","statuses_url":"https://api.github.com/repos/atom/atom/statuses/{sha}","languages_url":"https://api.github.com/repos/atom/atom/languages","stargazers_url":"https://api.github.com/repos/atom/atom/stargazers","contributors_url":"https://api.github.com/repos/atom/atom/contributors","subscribers_url":"https://api.github.com/repos/atom/atom/subscribers","subscription_url":"https://api.github.com/repos/atom/atom/subscription","commits_url":"https://api.github.com/repos/atom/atom/commits{/sha}","git_commits_url":"https://api.github.com/repos/atom/atom/git/commits{/sha}","comments_url":"https://api.github.com/repos/atom/atom/comments{/number}","issue_comment_url":"https://api.github.com/repos/atom/atom/issues/comments{/number}","contents_url":"https://api.github.com/repos/atom/atom/contents/{+path}","compare_url":"https://api.github.com/repos/atom/atom/compare/{base}...{head}","merges_url":"https://api.github.com/repos/atom/atom/merges","archive_url":"https://api.github.com/repos/atom/atom/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/atom/atom/downloads","issues_url":"https://api.github.com/repos/atom/atom/issues{/number}","pulls_url":"https://api.github.com/repos/atom/atom/pulls{/number}","milestones_url":"https://api.github.com/repos/atom/atom/milestones{/number}","notifications_url":"https://api.github.com/repos/atom/atom/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/atom/atom/labels{/name}","releases_url":"https://api.github.com/repos/atom/atom/releases{/id}","created_at":"2012-01-20T18:18:21Z","updated_at":"2015-05-20T15:52:29Z","pushed_at":"2015-05-20T15:58:01Z","git_url":"git://github.com/atom/atom.git","ssh_url":"git@github.com:atom/atom.git","clone_url":"https://github.com/atom/atom.git","svn_url":"https://github.com/atom/atom","homepage":"https://atom.io","size":363388,"stargazers_count":15478,"watchers_count":15478,"language":"CoffeeScript","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2719,"mirror_url":null,"open_issues_count":1087,"forks":2719,"open_issues":1087,"watchers":15478,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/atom/atom/pulls/6757"},"html":{"href":"https://github.com/atom/atom/pull/6757"},"issue":{"href":"https://api.github.com/repos/atom/atom/issues/6757"},"comments":{"href":"https://api.github.com/repos/atom/atom/issues/6757/comments"},"review_comments":{"href":"https://api.github.com/repos/atom/atom/pulls/6757/comments"},"review_comment":{"href":"https://api.github.com/repos/atom/atom/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/atom/atom/pulls/6757/commits"},"statuses":{"href":"https://api.github.com/repos/atom/atom/statuses/a109b3811c61f6115a398a757f09feae648f3eb7"}}}}