{"action":"created","issue":{"url":"https://api.github.com/repos/hankcs/HanLP/issues/18","labels_url":"https://api.github.com/repos/hankcs/HanLP/issues/18/labels{/name}","comments_url":"https://api.github.com/repos/hankcs/HanLP/issues/18/comments","events_url":"https://api.github.com/repos/hankcs/HanLP/issues/18/events","html_url":"https://github.com/hankcs/HanLP/issues/18","id":78352987,"number":18,"title":"CRF分词模型的训练 和 CRF分词视标点为新词问题","user":{"login":"Renzhh","id":8511776,"avatar_url":"https://avatars.githubusercontent.com/u/8511776?v=3","gravatar_id":"","url":"https://api.github.com/users/Renzhh","html_url":"https://github.com/Renzhh","followers_url":"https://api.github.com/users/Renzhh/followers","following_url":"https://api.github.com/users/Renzhh/following{/other_user}","gists_url":"https://api.github.com/users/Renzhh/gists{/gist_id}","starred_url":"https://api.github.com/users/Renzhh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Renzhh/subscriptions","organizations_url":"https://api.github.com/users/Renzhh/orgs","repos_url":"https://api.github.com/users/Renzhh/repos","events_url":"https://api.github.com/users/Renzhh/events{/privacy}","received_events_url":"https://api.github.com/users/Renzhh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-05-20T03:34:51Z","updated_at":"2015-05-20T04:01:44Z","closed_at":null,"body":"CRF分词模型使用的是什么语料？以及-f, -c参数的选取。只能通过不断的测试，选取经过测试选取效果好的？\r\n\r\n另，经使用1.2.2版的HanLP和data，测试CRF分词，测试数据如下：\r\n\r\n    \"《夜晚的骰子》通过描述浅草的舞女在暗夜中扔骰子的情景,寄托了作者对庶民生活区的情感\",\r\n    \"这个像是真的[委屈]前面那个打扮太江户了，一点不上品...@hankcs\", \r\n关闭词性标注的分词效果如下：\r\n\r\n    [《, 夜晚, 的, 骰子, 》, 通过, 描述, 浅草, 的, 舞女, 在, 暗夜, 中, 扔, 骰子, 的, 情景, ,, 寄托, 了, 作者, 对, 庶民, 生活区, 的, 情感]\r\n    [这个, 像, 是, 真的, [, 委屈, ], 前面, 那个, 打扮, 太, 江户, 了, ，, 一点, 不, 上品, ., ., ., @, hankcs]\r\n再查阅词典CoreNatureDictionary.txt，不存在其中的标注为null词性的词视为新词，以上两例句识别出的新词分别为：\r\n\r\n    识别到新词：,(wc: 1)\r\n    识别到新词：@(wc: 1), hankcs(wc: 1), [(wc: 1), ](wc: 1), .(wc: 3)\r\n整体效果不错，诸如\"hankcs\"词识别出了，但像\",\", \"@\", \"[\", \"]\", \".\"等标点符号也被视为新词。能够在CRF分词之后加入停用词之类的做法，过滤掉此类的词呢？\r\n\r\n谢谢！"},"comment":{"url":"https://api.github.com/repos/hankcs/HanLP/issues/comments/103748458","html_url":"https://github.com/hankcs/HanLP/issues/18#issuecomment-103748458","issue_url":"https://api.github.com/repos/hankcs/HanLP/issues/18","id":103748458,"user":{"login":"hankcs","id":5326890,"avatar_url":"https://avatars.githubusercontent.com/u/5326890?v=3","gravatar_id":"","url":"https://api.github.com/users/hankcs","html_url":"https://github.com/hankcs","followers_url":"https://api.github.com/users/hankcs/followers","following_url":"https://api.github.com/users/hankcs/following{/other_user}","gists_url":"https://api.github.com/users/hankcs/gists{/gist_id}","starred_url":"https://api.github.com/users/hankcs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hankcs/subscriptions","organizations_url":"https://api.github.com/users/hankcs/orgs","repos_url":"https://api.github.com/users/hankcs/repos","events_url":"https://api.github.com/users/hankcs/events{/privacy}","received_events_url":"https://api.github.com/users/hankcs/received_events","type":"User","site_admin":false},"created_at":"2015-05-20T04:01:44Z","updated_at":"2015-05-20T04:01:44Z","body":"感谢建议。\r\n\r\n0. 2014人民日报语料库\r\n0. 调参是个苦活，硬件条件有限，训练一个模型得12个小时，所以暂未考虑。如果你有更好的机器，可以自行训练，只要输出文本形式的模型，HanLP就可以读取。\r\n0. 用词性是否为null来判断新词的确很朴素。我的确在构想提供更实用的API，但是我倾向于只提供接口，向用户隐藏内部的细节（基于CRF？基于TnT？基于信息熵？）。内部的实现当然会考虑标点的问题，考虑排重，考虑用户词典，过滤停用词、过滤英文、数词、日期、人名……所以这套东西虽然没什么难度，但是还是有点工作量的，大约会在七月份去做掉。在此之前，欢迎继续讨论，提供宝贵意见。"}}