{"action":"created","issue":{"url":"https://api.github.com/repos/markdown-it/markdown-it/issues/37","labels_url":"https://api.github.com/repos/markdown-it/markdown-it/issues/37/labels{/name}","comments_url":"https://api.github.com/repos/markdown-it/markdown-it/issues/37/comments","events_url":"https://api.github.com/repos/markdown-it/markdown-it/issues/37/events","html_url":"https://github.com/markdown-it/markdown-it/issues/37","id":54118488,"number":37,"title":"Token stream redesign","user":{"login":"puzrin","id":319465,"avatar_url":"https://avatars.githubusercontent.com/u/319465?v=3","gravatar_id":"","url":"https://api.github.com/users/puzrin","html_url":"https://github.com/puzrin","followers_url":"https://api.github.com/users/puzrin/followers","following_url":"https://api.github.com/users/puzrin/following{/other_user}","gists_url":"https://api.github.com/users/puzrin/gists{/gist_id}","starred_url":"https://api.github.com/users/puzrin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/puzrin/subscriptions","organizations_url":"https://api.github.com/users/puzrin/orgs","repos_url":"https://api.github.com/users/puzrin/repos","events_url":"https://api.github.com/users/puzrin/events{/privacy}","received_events_url":"https://api.github.com/users/puzrin/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/markdown-it/markdown-it/labels/dev+memo","name":"dev memo","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-01-12T22:46:11Z","updated_at":"2015-01-13T20:49:53Z","closed_at":null,"body":"Goals:\r\n\r\n- unify renderers as much as possible\r\n- simplify AST build\r\n- unify attributes rendering\r\n\r\n---\r\n\r\n\r\n```yaml\r\ntoken:\r\n  type: link_open # ? may be \"link\", but that can make pairs scan more complex.\r\n  tag: a\r\n\r\n  # join as -1 | 0 | 1 ?\r\n  single: false # true for pairs\r\n  open: true # opening / closing tag\r\n\r\n  map: [ line_begin, line_end ] # may be sometime [ line_begin, line_end, col_begin ]\r\n\r\n  # only static things, with standard processing (html escape)\r\n  # may be with boolean attrs\r\n  # (?) complex things like multiple classes - now as single string, words separated by space\r\n  attrs:\r\n    - [ href, data ]\r\n    - [ target, _blank ]\r\n    - [ title, 'titke text' ]\r\n    - [ alt, 'alt text' ]\r\n    - [ id, '#...' ]\r\n\r\n  markup: string # optional, describes used markup (bullet char, emphasis marker, hr markup)\r\n\r\n  # self-closing tokens (fence, code)\r\n  content: unparsed content\r\n  fence_offset: 0\r\n  info: ''\r\n\r\n  # (?) link, image only (or via attributes?)\r\n  destination: \"http://...\"\r\n\r\n  # heading only\r\n  heading_level: 3\r\n\r\n  # paragraph only\r\n  tight: false # true - member of tight list, should be hidden\r\n\r\n  # inline container\r\n  children: [...] # inline tags \r\n```\r\n\r\nThis ticket is for core devs only."},"comment":{"url":"https://api.github.com/repos/markdown-it/markdown-it/issues/comments/69817171","html_url":"https://github.com/markdown-it/markdown-it/issues/37#issuecomment-69817171","issue_url":"https://api.github.com/repos/markdown-it/markdown-it/issues/37","id":69817171,"user":{"login":"marijnh","id":144427,"avatar_url":"https://avatars.githubusercontent.com/u/144427?v=3","gravatar_id":"","url":"https://api.github.com/users/marijnh","html_url":"https://github.com/marijnh","followers_url":"https://api.github.com/users/marijnh/followers","following_url":"https://api.github.com/users/marijnh/following{/other_user}","gists_url":"https://api.github.com/users/marijnh/gists{/gist_id}","starred_url":"https://api.github.com/users/marijnh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marijnh/subscriptions","organizations_url":"https://api.github.com/users/marijnh/orgs","repos_url":"https://api.github.com/users/marijnh/repos","events_url":"https://api.github.com/users/marijnh/events{/privacy}","received_events_url":"https://api.github.com/users/marijnh/received_events","type":"User","site_admin":false},"created_at":"2015-01-13T20:49:53Z","updated_at":"2015-01-13T20:49:53Z","body":"I don't care much about whether the format is an AST or a token stream -- I'm defining my own more abstract tree anyway, and both are easy to convert from and to. I also am not interested in perfect restoration of the text format, but I'd like to make a reasonable effort there and not change things without a good reason to do so.\r\n\r\nI guess your parser does at some point separate out the text that makes for the content of a block, in order to be able to run further block and inline parser on it. Would it be viable to save the text at this point? (I'm also perfectly okay with having an optional hook mechanism that allows me to register a callback that adds stuff to token objects, so that users who don't need this data don't have to pay the price.)"}}