{"action":"created","comment":{"url":"https://api.github.com/repos/apache/tika/pulls/comments/44430770","id":44430770,"diff_hunk":"@@ -0,0 +1,53 @@\n+/*\n+ * Licensed to the Apache Software Foundation (ASF) under one or more\n+ * contributor license agreements.  See the NOTICE file distributed with\n+ * this work for additional information regarding copyright owlocationNameEntitieship.\n+ * The ASF licenses this file to You under the Apache License, Version 2.0\n+ * (the \"License\"); you may not use this file except in compliance with\n+ * the License.  You may obtain a copy of the License at\n+ *\n+ *     http://www.apache.org/licenses/LICENSE-2.0\n+ *\n+ * Unless required by applicable law or agreed to in writing, software\n+ * distributed under the License is distributed on an \"AS IS\" BASIS,\n+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n+ * See the License for the specific language governing permissions and\n+ * limitations under the License.\n+ */\n+package org.apache.tika.parser.ner.regex;\n+\n+import org.apache.tika.Tika;\n+import org.apache.tika.config.TikaConfig;\n+import org.apache.tika.metadata.Metadata;\n+import org.apache.tika.parser.ner.NamedEntityParser;\n+import org.junit.Test;\n+\n+import java.io.ByteArrayInputStream;\n+import java.nio.charset.StandardCharsets;\n+import java.util.Arrays;\n+import java.util.HashSet;\n+import java.util.Set;\n+\n+import static org.junit.Assert.*;","path":"tika-parsers/src/test/java/org/apache/tika/parser/ner/regex/RegexNERecogniserTest.java","position":31,"original_position":31,"commit_id":"7e6b43c83ec6cdd35ea258f52c0110ba986c82b3","original_commit_id":"7e6b43c83ec6cdd35ea258f52c0110ba986c82b3","user":{"login":"chrismattmann","id":395887,"avatar_url":"https://avatars.githubusercontent.com/u/395887?v=3","gravatar_id":"","url":"https://api.github.com/users/chrismattmann","html_url":"https://github.com/chrismattmann","followers_url":"https://api.github.com/users/chrismattmann/followers","following_url":"https://api.github.com/users/chrismattmann/following{/other_user}","gists_url":"https://api.github.com/users/chrismattmann/gists{/gist_id}","starred_url":"https://api.github.com/users/chrismattmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrismattmann/subscriptions","organizations_url":"https://api.github.com/users/chrismattmann/orgs","repos_url":"https://api.github.com/users/chrismattmann/repos","events_url":"https://api.github.com/users/chrismattmann/events{/privacy}","received_events_url":"https://api.github.com/users/chrismattmann/received_events","type":"User","site_admin":false},"body":"no star imports","created_at":"2015-11-10T16:45:06Z","updated_at":"2015-11-10T16:45:06Z","html_url":"https://github.com/apache/tika/pull/61#discussion_r44430770","pull_request_url":"https://api.github.com/repos/apache/tika/pulls/61","_links":{"self":{"href":"https://api.github.com/repos/apache/tika/pulls/comments/44430770"},"html":{"href":"https://github.com/apache/tika/pull/61#discussion_r44430770"},"pull_request":{"href":"https://api.github.com/repos/apache/tika/pulls/61"}}},"pull_request":{"url":"https://api.github.com/repos/apache/tika/pulls/61","id":49341905,"html_url":"https://github.com/apache/tika/pull/61","diff_url":"https://github.com/apache/tika/pull/61.diff","patch_url":"https://github.com/apache/tika/pull/61.patch","issue_url":"https://api.github.com/repos/apache/tika/issues/61","number":61,"state":"open","locked":false,"title":"NamedEntityParser","user":{"login":"thammegowda","id":1865964,"avatar_url":"https://avatars.githubusercontent.com/u/1865964?v=3","gravatar_id":"","url":"https://api.github.com/users/thammegowda","html_url":"https://github.com/thammegowda","followers_url":"https://api.github.com/users/thammegowda/followers","following_url":"https://api.github.com/users/thammegowda/following{/other_user}","gists_url":"https://api.github.com/users/thammegowda/gists{/gist_id}","starred_url":"https://api.github.com/users/thammegowda/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thammegowda/subscriptions","organizations_url":"https://api.github.com/users/thammegowda/orgs","repos_url":"https://api.github.com/users/thammegowda/repos","events_url":"https://api.github.com/users/thammegowda/events{/privacy}","received_events_url":"https://api.github.com/users/thammegowda/received_events","type":"User","site_admin":false},"body":"Added NamedEntityParser that supports loading of different NER implementations at runtime.\r\nThe default NER implementation based on OpenNLP is supplied.\r\n\r\n~~Another implementation based on StanfordCoreNLP is located [here](https://github.com/thammegowda/tika-ner-corenlp/blob/master/src/test/java/edu/usc/cs/ir/tika/ner/NamedEntityParserTest.java) This is GNU GPL 3, So kept separate.~~ See UPDATE 2 below\r\n\r\n@chrismattmann This is not  100% complete, here are few TODOs : \r\n1. The NER implementing class name needs is to be read from tika config if possible/available. Currently relying on Java Properties. Please suggest me on how to resolve [this todo](https://github.com/thammegowda/tika/commit/e96da2bc28d5eef81d034e39eb05099ed5d38ac1#diff-3a416957f7629e40c8fe5e8fd6c577a3R70)\r\nEDIT : ~~2. Looking for a best way to read parsed text from non text streams within the NamedEntityParser (not sure if a parser can read output of previous parsers like html or pdf). Please suggest me on how to resolve [this todo](https://github.com/thammegowda/tika/commit/e96da2bc28d5eef81d034e39eb05099ed5d38ac1#diff-3a416957f7629e40c8fe5e8fd6c577a3R91)~~ Using secondary parser to get text content\r\n\r\nUPDATE : 1. [Added Regex Based NER ](https://github.com/thammegowda/tika/commit/ea7871bd4afae7d18e500ffc285e58afd08f5e86). Though this can recognize much more patterns than names, _(I am using it for recognising weapon names and weapon types )_\r\n\r\nUPDATE 2 : [Added Core NLP NER with runtime class binding](https://github.com/thammegowda/tika/commit/084985b3612438e9ca7107fecdffd67757d04d10) this one is still using java binding instead of command invocation, because :\r\n  +  The commandline binding is not portable across environments (or need to maintain those many ports)\r\n  +  setup overead in distribted environment like hadoop.\r\n","created_at":"2015-10-31T04:53:44Z","updated_at":"2015-11-10T16:45:06Z","closed_at":null,"merged_at":null,"merge_commit_sha":"36271c3af43c93ea06b9bbb30a31fb1aac6cee3d","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/tika/pulls/61/commits","review_comments_url":"https://api.github.com/repos/apache/tika/pulls/61/comments","review_comment_url":"https://api.github.com/repos/apache/tika/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/tika/issues/61/comments","statuses_url":"https://api.github.com/repos/apache/tika/statuses/7e6b43c83ec6cdd35ea258f52c0110ba986c82b3","head":{"label":"thammegowda:trunk","ref":"trunk","sha":"7e6b43c83ec6cdd35ea258f52c0110ba986c82b3","user":{"login":"thammegowda","id":1865964,"avatar_url":"https://avatars.githubusercontent.com/u/1865964?v=3","gravatar_id":"","url":"https://api.github.com/users/thammegowda","html_url":"https://github.com/thammegowda","followers_url":"https://api.github.com/users/thammegowda/followers","following_url":"https://api.github.com/users/thammegowda/following{/other_user}","gists_url":"https://api.github.com/users/thammegowda/gists{/gist_id}","starred_url":"https://api.github.com/users/thammegowda/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thammegowda/subscriptions","organizations_url":"https://api.github.com/users/thammegowda/orgs","repos_url":"https://api.github.com/users/thammegowda/repos","events_url":"https://api.github.com/users/thammegowda/events{/privacy}","received_events_url":"https://api.github.com/users/thammegowda/received_events","type":"User","site_admin":false},"repo":{"id":44792163,"name":"tika","full_name":"thammegowda/tika","owner":{"login":"thammegowda","id":1865964,"avatar_url":"https://avatars.githubusercontent.com/u/1865964?v=3","gravatar_id":"","url":"https://api.github.com/users/thammegowda","html_url":"https://github.com/thammegowda","followers_url":"https://api.github.com/users/thammegowda/followers","following_url":"https://api.github.com/users/thammegowda/following{/other_user}","gists_url":"https://api.github.com/users/thammegowda/gists{/gist_id}","starred_url":"https://api.github.com/users/thammegowda/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thammegowda/subscriptions","organizations_url":"https://api.github.com/users/thammegowda/orgs","repos_url":"https://api.github.com/users/thammegowda/repos","events_url":"https://api.github.com/users/thammegowda/events{/privacy}","received_events_url":"https://api.github.com/users/thammegowda/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/thammegowda/tika","description":"Mirror of Apache Tika","fork":true,"url":"https://api.github.com/repos/thammegowda/tika","forks_url":"https://api.github.com/repos/thammegowda/tika/forks","keys_url":"https://api.github.com/repos/thammegowda/tika/keys{/key_id}","collaborators_url":"https://api.github.com/repos/thammegowda/tika/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/thammegowda/tika/teams","hooks_url":"https://api.github.com/repos/thammegowda/tika/hooks","issue_events_url":"https://api.github.com/repos/thammegowda/tika/issues/events{/number}","events_url":"https://api.github.com/repos/thammegowda/tika/events","assignees_url":"https://api.github.com/repos/thammegowda/tika/assignees{/user}","branches_url":"https://api.github.com/repos/thammegowda/tika/branches{/branch}","tags_url":"https://api.github.com/repos/thammegowda/tika/tags","blobs_url":"https://api.github.com/repos/thammegowda/tika/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/thammegowda/tika/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/thammegowda/tika/git/refs{/sha}","trees_url":"https://api.github.com/repos/thammegowda/tika/git/trees{/sha}","statuses_url":"https://api.github.com/repos/thammegowda/tika/statuses/{sha}","languages_url":"https://api.github.com/repos/thammegowda/tika/languages","stargazers_url":"https://api.github.com/repos/thammegowda/tika/stargazers","contributors_url":"https://api.github.com/repos/thammegowda/tika/contributors","subscribers_url":"https://api.github.com/repos/thammegowda/tika/subscribers","subscription_url":"https://api.github.com/repos/thammegowda/tika/subscription","commits_url":"https://api.github.com/repos/thammegowda/tika/commits{/sha}","git_commits_url":"https://api.github.com/repos/thammegowda/tika/git/commits{/sha}","comments_url":"https://api.github.com/repos/thammegowda/tika/comments{/number}","issue_comment_url":"https://api.github.com/repos/thammegowda/tika/issues/comments{/number}","contents_url":"https://api.github.com/repos/thammegowda/tika/contents/{+path}","compare_url":"https://api.github.com/repos/thammegowda/tika/compare/{base}...{head}","merges_url":"https://api.github.com/repos/thammegowda/tika/merges","archive_url":"https://api.github.com/repos/thammegowda/tika/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/thammegowda/tika/downloads","issues_url":"https://api.github.com/repos/thammegowda/tika/issues{/number}","pulls_url":"https://api.github.com/repos/thammegowda/tika/pulls{/number}","milestones_url":"https://api.github.com/repos/thammegowda/tika/milestones{/number}","notifications_url":"https://api.github.com/repos/thammegowda/tika/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/thammegowda/tika/labels{/name}","releases_url":"https://api.github.com/repos/thammegowda/tika/releases{/id}","created_at":"2015-10-23T05:13:55Z","updated_at":"2015-10-23T05:14:03Z","pushed_at":"2015-11-10T01:41:12Z","git_url":"git://github.com/thammegowda/tika.git","ssh_url":"git@github.com:thammegowda/tika.git","clone_url":"https://github.com/thammegowda/tika.git","svn_url":"https://github.com/thammegowda/tika","homepage":null,"size":83750,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"trunk"}},"base":{"label":"apache:trunk","ref":"trunk","sha":"dc8067d703410598a41f197167ef912c7845ee12","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":206427,"name":"tika","full_name":"apache/tika","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/tika","description":"Mirror of Apache Tika","fork":false,"url":"https://api.github.com/repos/apache/tika","forks_url":"https://api.github.com/repos/apache/tika/forks","keys_url":"https://api.github.com/repos/apache/tika/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/tika/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/tika/teams","hooks_url":"https://api.github.com/repos/apache/tika/hooks","issue_events_url":"https://api.github.com/repos/apache/tika/issues/events{/number}","events_url":"https://api.github.com/repos/apache/tika/events","assignees_url":"https://api.github.com/repos/apache/tika/assignees{/user}","branches_url":"https://api.github.com/repos/apache/tika/branches{/branch}","tags_url":"https://api.github.com/repos/apache/tika/tags","blobs_url":"https://api.github.com/repos/apache/tika/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/tika/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/tika/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/tika/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/tika/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/tika/languages","stargazers_url":"https://api.github.com/repos/apache/tika/stargazers","contributors_url":"https://api.github.com/repos/apache/tika/contributors","subscribers_url":"https://api.github.com/repos/apache/tika/subscribers","subscription_url":"https://api.github.com/repos/apache/tika/subscription","commits_url":"https://api.github.com/repos/apache/tika/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/tika/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/tika/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/tika/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/tika/contents/{+path}","compare_url":"https://api.github.com/repos/apache/tika/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/tika/merges","archive_url":"https://api.github.com/repos/apache/tika/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/tika/downloads","issues_url":"https://api.github.com/repos/apache/tika/issues{/number}","pulls_url":"https://api.github.com/repos/apache/tika/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/tika/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/tika/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/tika/labels{/name}","releases_url":"https://api.github.com/repos/apache/tika/releases{/id}","created_at":"2009-05-21T02:12:11Z","updated_at":"2015-11-10T13:24:56Z","pushed_at":"2015-11-10T16:19:12Z","git_url":"git://github.com/apache/tika.git","ssh_url":"git@github.com:apache/tika.git","clone_url":"https://github.com/apache/tika.git","svn_url":"https://github.com/apache/tika","homepage":null,"size":100434,"stargazers_count":240,"watchers_count":240,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":149,"mirror_url":"git://git.apache.org/tika.git","open_issues_count":3,"forks":149,"open_issues":3,"watchers":240,"default_branch":"trunk"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/tika/pulls/61"},"html":{"href":"https://github.com/apache/tika/pull/61"},"issue":{"href":"https://api.github.com/repos/apache/tika/issues/61"},"comments":{"href":"https://api.github.com/repos/apache/tika/issues/61/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/tika/pulls/61/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/tika/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/tika/pulls/61/commits"},"statuses":{"href":"https://api.github.com/repos/apache/tika/statuses/7e6b43c83ec6cdd35ea258f52c0110ba986c82b3"}}}}