{"action":"created","issue":{"url":"https://api.github.com/repos/BioJulia/Bio.jl/issues/80","labels_url":"https://api.github.com/repos/BioJulia/Bio.jl/issues/80/labels{/name}","comments_url":"https://api.github.com/repos/BioJulia/Bio.jl/issues/80/comments","events_url":"https://api.github.com/repos/BioJulia/Bio.jl/issues/80/events","html_url":"https://github.com/BioJulia/Bio.jl/pull/80","id":101742018,"number":80,"title":"Tokenizer tool","user":{"login":"Ward9250","id":3399309,"avatar_url":"https://avatars.githubusercontent.com/u/3399309?v=3","gravatar_id":"","url":"https://api.github.com/users/Ward9250","html_url":"https://github.com/Ward9250","followers_url":"https://api.github.com/users/Ward9250/followers","following_url":"https://api.github.com/users/Ward9250/following{/other_user}","gists_url":"https://api.github.com/users/Ward9250/gists{/gist_id}","starred_url":"https://api.github.com/users/Ward9250/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Ward9250/subscriptions","organizations_url":"https://api.github.com/users/Ward9250/orgs","repos_url":"https://api.github.com/users/Ward9250/repos","events_url":"https://api.github.com/users/Ward9250/events{/privacy}","received_events_url":"https://api.github.com/users/Ward9250/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/BioJulia/Bio.jl/labels/WIP","name":"WIP","color":"e11d21"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-08-18T20:29:40Z","updated_at":"2015-09-09T01:26:00Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/BioJulia/Bio.jl/pulls/80","html_url":"https://github.com/BioJulia/Bio.jl/pull/80","diff_url":"https://github.com/BioJulia/Bio.jl/pull/80.diff","patch_url":"https://github.com/BioJulia/Bio.jl/pull/80.patch"},"body":"This was originally part of the Phylo module, but I’ve decided it should be rehired in the tools.jl file or Bio.jl and treated as a general utility available to Bio.jl users/modules as I’ve found several uses for it, first for tokenising Newick, and now doing the same for CIGAR strings. "},"comment":{"url":"https://api.github.com/repos/BioJulia/Bio.jl/issues/comments/138746031","html_url":"https://github.com/BioJulia/Bio.jl/pull/80#issuecomment-138746031","issue_url":"https://api.github.com/repos/BioJulia/Bio.jl/issues/80","id":138746031,"user":{"login":"dcjones","id":210361,"avatar_url":"https://avatars.githubusercontent.com/u/210361?v=3","gravatar_id":"","url":"https://api.github.com/users/dcjones","html_url":"https://github.com/dcjones","followers_url":"https://api.github.com/users/dcjones/followers","following_url":"https://api.github.com/users/dcjones/following{/other_user}","gists_url":"https://api.github.com/users/dcjones/gists{/gist_id}","starred_url":"https://api.github.com/users/dcjones/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dcjones/subscriptions","organizations_url":"https://api.github.com/users/dcjones/orgs","repos_url":"https://api.github.com/users/dcjones/repos","events_url":"https://api.github.com/users/dcjones/events{/privacy}","received_events_url":"https://api.github.com/users/dcjones/received_events","type":"User","site_admin":false},"created_at":"2015-09-09T01:25:59Z","updated_at":"2015-09-09T01:25:59Z","body":"You can tokenize newick with ragel. You just can easily parse it completely, since matching parenthesis is context free and not regular. \r\n\r\nHere's a basic newick tokenizer:\r\n```julia\r\n\r\nimport Bio\r\nimport Bio.Ragel\r\nusing BufferedStreams, Switch\r\n\r\n\r\nimmutable Newick <: Bio.FileFormat end\r\n\r\n\r\ntype NewickTree\r\n    # ...\r\nend\r\n\r\n\r\n%%{\r\n    machine newick;\r\n\r\n    action anchor { Ragel.@anchor! }\r\n    action token {\r\n        Ragel.@copy_from_anchor!(token)\r\n        println(token)\r\n    }\r\n\r\n    length = '-'? digit+ ('.' digit*)?;\r\n    unquoted_label = [A-Za-z_\\-] [A-Za-z0-9_\\-]*;\r\n    quoted_label = '\"' [^\"]* '\"';\r\n    token = ('(' | ')' | ',' | ':' | ';' | length | unquoted_label | quoted_label) >anchor %token;\r\n\r\n    main := space* (token space*)**;\r\n}%%\r\n\r\n\r\n%% write data;\r\n\r\n\r\ntype NewickParser <: Ragel.AbstractParser\r\n    state::Ragel.State\r\n\r\n    function NewickParser(input::BufferedInputStream)\r\n        %% write init;\r\n\r\n        return new(Ragel.State(cs, input))\r\n    end\r\nend\r\n\r\n\r\nfunction Base.open(stream::BufferedInputStream, ::Type{Newick})\r\n    return NewickParser(stream)\r\nend\r\n\r\n\r\nRagel.@generate_read_fuction(\"newick\", NewickParser, NewickTree,\r\n    begin\r\n        token = Bio.StringField()\r\n\r\n        %% write exec;\r\n    end)\r\n\r\n\r\nexample = \"\"\"\r\n    (\r\n    (raccoon:19.19959,bear:6.80041):0.84600,\r\n    (\r\n        (sea_lion:11.99700, seal:12.00300):7.52973,\r\n        (\r\n            (monkey:100.85930,cat:47.14069):20.59201, \r\n            weasel:18.87953\r\n        ):2.09460\r\n    ):3.87382,\r\n    dog:25.46154\r\n);\r\n\"\"\"\r\n\r\nparser = open(example.data, Newick)\r\ntree = NewickTree()\r\nread!(parser, tree)\r\n```\r\n\r\nWhich prints:\r\n```\r\n(\r\n(\r\nraccoon\r\n:\r\n19.19959\r\n,\r\nbear\r\n:\r\n6.80041\r\n)\r\n:\r\n0.84600\r\n,\r\n(\r\n(\r\nsea_lion\r\n:\r\n11.99700\r\n,\r\nseal\r\n:\r\n12.00300\r\n)\r\n:\r\n7.52973\r\n,\r\n(\r\n(\r\nmonkey\r\n:\r\n100.85930\r\n,\r\ncat\r\n:\r\n47.14069\r\n)\r\n:\r\n20.59201\r\n,\r\nweasel\r\n:\r\n18.87953\r\n)\r\n:\r\n2.09460\r\n)\r\n:\r\n3.87382\r\n,\r\ndog\r\n:\r\n25.46154\r\n)\r\n;\r\n```\r\n\r\nFor a complete parser, you just need to do something with the tokens beyond printing them.\r\n"}}