{"push_id":694587453,"size":2,"distinct_size":2,"ref":"refs/heads/4.0.7","head":"0a39552b24c6a64f797c74b15f7a66a675f6e59a","before":"6bec019104cba032c74cbcd5adb3446544eb0479","commits":[{"sha":"c3c5bf2af1cc0408e26f68c838bd087cf7edce70","author":{"email":"9039cdb0d8db19b5176362cec217d50253d15cba@163.com","name":"yin-bp"},"message":"增加通用分词工具类，输入文本串，定义变量前缀和后缀，返回相应的token列表，实例如下：\n\nString tokenpre = \"#inlcude(\"; \n\nchar tokenend = ')'; \nString content = \"#inlcude(head.html)\"; \nList<GrammarToken> tokens = TextGrammarParser.parser(content, tokenpre,\ntokenend); \nSystem.out.println(content+\"分词结果：\\r\\n\"); \nSystem.out.println(tokens); \n\ncontent = \"asdfasdf#inlcude(#inlcude(head.html)\"; \ntokens = TextGrammarParser.parser(content, tokenpre, tokenend); \nSystem.out.println(content+\"分词结果：\\r\\n\"); \nSystem.out.println(tokens); \n\ncontent = \"asdfasdf#inlcude(#inlcude(head.html))\"; \ntokens = TextGrammarParser.parser(content, tokenpre, tokenend); \nSystem.out.println(content+\"分词结果：\\r\\n\"); \nSystem.out.println(tokens); \n\ncontent = \"asdfasdf#inlcude(#inlcude(head.html\"; \ntokens = TextGrammarParser.parser(content, tokenpre, tokenend); \nSystem.out.println(content+\"分词结果：\\r\\n\"); \nSystem.out.println(tokens); \n\ncontent = \"#inlcude()\"; \ntokens = TextGrammarParser.parser(content, tokenpre, tokenend); \nSystem.out.println(content+\"分词结果：\\r\\n\"); \nSystem.out.println(tokens);\n\n\n#inlcude(head.html)分词结果： \n\n[text=head.html|position=0|type=结构变量 \n] \nasdfasdf#inlcude(#inlcude(head.html)分词结果： \n\n[text=asdfasdf#inlcude(|position=0|type=普通文本 \n, text=head.html|position=1|type=结构变量 \n] \nasdfasdf#inlcude(#inlcude(head.html))分词结果： \n\n[text=asdfasdf#inlcude(|position=0|type=普通文本 \n, text=head.html|position=1|type=结构变量 \n, text=)|position=2|type=普通文本 \n] \nasdfasdf#inlcude(#inlcude(head.html分词结果： \n\n[text=asdfasdf#inlcude(#inlcude(head.html|position=0|type=普通文本 \n] \n#inlcude()分词结果： \n\n[text=#inlcude()|position=0|type=普通文本 \n]","distinct":true,"url":"https://api.github.com/repos/bbossgroups/bbossgroups-3.5/commits/c3c5bf2af1cc0408e26f68c838bd087cf7edce70"},{"sha":"0a39552b24c6a64f797c74b15f7a66a675f6e59a","author":{"email":"9039cdb0d8db19b5176362cec217d50253d15cba@163.com","name":"yin-bp"},"message":"增加通用分词工具类，输入文本串，定义变量前缀和后缀，返回相应的token列表，实例如下：\n\nString tokenpre = \"#inlcude(\"; \n\nchar tokenend = ')'; \nString content = \"#inlcude(head.html)\"; \nList<GrammarToken> tokens = TextGrammarParser.parser(content, tokenpre,\ntokenend); \nSystem.out.println(content+\"分词结果：\\r\\n\"); \nSystem.out.println(tokens); \n\ncontent = \"asdfasdf#inlcude(#inlcude(head.html)\"; \ntokens = TextGrammarParser.parser(content, tokenpre, tokenend); \nSystem.out.println(content+\"分词结果：\\r\\n\"); \nSystem.out.println(tokens); \n\ncontent = \"asdfasdf#inlcude(#inlcude(head.html))\"; \ntokens = TextGrammarParser.parser(content, tokenpre, tokenend); \nSystem.out.println(content+\"分词结果：\\r\\n\"); \nSystem.out.println(tokens); \n\ncontent = \"asdfasdf#inlcude(#inlcude(head.html\"; \ntokens = TextGrammarParser.parser(content, tokenpre, tokenend); \nSystem.out.println(content+\"分词结果：\\r\\n\"); \nSystem.out.println(tokens); \n\ncontent = \"#inlcude()\"; \ntokens = TextGrammarParser.parser(content, tokenpre, tokenend); \nSystem.out.println(content+\"分词结果：\\r\\n\"); \nSystem.out.println(tokens);\n\n\n#inlcude(head.html)分词结果： \n\n[text=head.html|position=0|type=结构变量 \n] \nasdfasdf#inlcude(#inlcude(head.html)分词结果： \n\n[text=asdfasdf#inlcude(|position=0|type=普通文本 \n, text=head.html|position=1|type=结构变量 \n] \nasdfasdf#inlcude(#inlcude(head.html))分词结果： \n\n[text=asdfasdf#inlcude(|position=0|type=普通文本 \n, text=head.html|position=1|type=结构变量 \n, text=)|position=2|type=普通文本 \n] \nasdfasdf#inlcude(#inlcude(head.html分词结果： \n\n[text=asdfasdf#inlcude(#inlcude(head.html|position=0|type=普通文本 \n] \n#inlcude()分词结果： \n\n[text=#inlcude()|position=0|type=普通文本 \n]","distinct":true,"url":"https://api.github.com/repos/bbossgroups/bbossgroups-3.5/commits/0a39552b24c6a64f797c74b15f7a66a675f6e59a"}]}