{"action":"opened","number":109,"pull_request":{"url":"https://api.github.com/repos/allenai/nlpstack/pulls/109","id":31307008,"html_url":"https://github.com/allenai/nlpstack/pull/109","diff_url":"https://github.com/allenai/nlpstack/pull/109.diff","patch_url":"https://github.com/allenai/nlpstack/pull/109.patch","issue_url":"https://api.github.com/repos/allenai/nlpstack/issues/109","number":109,"state":"open","locked":false,"title":"Making Verbnet feature more granular","user":{"login":"sbhaktha","id":5631150,"avatar_url":"https://avatars.githubusercontent.com/u/5631150?v=3","gravatar_id":"","url":"https://api.github.com/users/sbhaktha","html_url":"https://github.com/sbhaktha","followers_url":"https://api.github.com/users/sbhaktha/followers","following_url":"https://api.github.com/users/sbhaktha/following{/other_user}","gists_url":"https://api.github.com/users/sbhaktha/gists{/gist_id}","starred_url":"https://api.github.com/users/sbhaktha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sbhaktha/subscriptions","organizations_url":"https://api.github.com/users/sbhaktha/orgs","repos_url":"https://api.github.com/users/sbhaktha/repos","events_url":"https://api.github.com/users/sbhaktha/events{/privacy}","received_events_url":"https://api.github.com/users/sbhaktha/received_events","type":"User","site_admin":false},"body":"@mhrmm : Changes as discussed. Marginal improvement in scores.\r\nNote that based on the table at the top of this page: http://verbs.colorado.edu/~mpalmer/projects/verbnet.html#frames, you had suggested using the Frame Name and the Frame Syntax. For the e.g. in that table (hit-18.1), this is what the frame xml snippet looks like:\r\n\r\n```\r\n        <FRAME>\r\n            <DESCRIPTION descriptionNumber=\"0.2\" primary=\"NP V NP\" secondary=\"Basic Transitive\" xtag=\"0.2\"/>\r\n            <EXAMPLES>\r\n                <EXAMPLE>Paula hit the ball.</EXAMPLE>\r\n            </EXAMPLES>\r\n            <SYNTAX>\r\n                <NP value=\"Agent\">\r\n                    <SYNRESTRS/>\r\n                </NP>\r\n                <VERB/>\r\n                <NP value=\"Patient\">\r\n                    <SYNRESTRS/>\r\n                </NP>\r\n            </SYNTAX>\r\n...\r\n```\r\n\r\n`Basic Transitive` is the secondary name, and `NP V NP` is the primary frame name, which represents the syntax `Agent V Patient`. So I have used the primary frame names and the secondary frame names as the features. Since a class can have multiple frames, there can be multiple values (a set) coming from the same class for a given verb.\r\n\r\nThe `featuretaggers.config` (rename of the earlier `polyparser.config`) can be used to pass any config info for any other features, for e.g., the Brown clusters, which are not in the datastore yet.\r\n\r\nScores\r\n\r\nBaseline\r\n\r\n```\r\nUAS: 9084 / 10427 = 87.12%\r\nUAS (no punc): 8019 / 9130 = 87.83%\r\nLAS: 8663 / 10427 = 83.08%\r\nLAS (no punc): 7598 / 9130 = 83.22%\r\nPath Accuracy (labeled, full): 7371 / 10427 = 70.69%\r\nPath Accuracy (unlabeled, full): 7903 / 10427 = 75.79%\r\nPath Accuracy (labeled, ignorePunc): 6347 / 9130 = 69.52%\r\nPath Accuracy (unlabeled, ignorePunc): 6875 / 9130 = 75.30%\r\nParsed 1000 sentences in 16.3 seconds, an average of 61.4 sentences per second.\r\n```\r\n\r\n\r\nWith Verbnet\r\n\r\n```\r\nUAS: 9142 / 10427 = 87.68%\r\nUAS (no punc): 8059 / 9130 = 88.27%\r\nLAS: 8741 / 10427 = 83.83%\r\nLAS (no punc): 7659 / 9130 = 83.89%\r\nPath Accuracy (labeled, full): 7515 / 10427 = 72.07%\r\nPath Accuracy (unlabeled, full): 8007 / 10427 = 76.79%\r\nPath Accuracy (labeled, ignorePunc): 6465 / 9130 = 70.81%\r\nPath Accuracy (unlabeled, ignorePunc): 6954 / 9130 = 76.17%\r\nParsed 1000 sentences in 19.4 seconds, an average of 51.4 sentences per second.\r\n```\r\n\r\n\r\nWith Brown Clusters\r\n\r\n```\r\nUAS: 9154 / 10427 = 87.79%\r\nUAS (no punc): 8084 / 9130 = 88.54%\r\nLAS: 8780 / 10427 = 84.20%\r\nLAS (no punc): 7710 / 9130 = 84.45%\r\nPath Accuracy (labeled, full): 7505 / 10427 = 71.98%\r\nPath Accuracy (unlabeled, full): 8027 / 10427 = 76.98%\r\nPath Accuracy (labeled, ignorePunc): 6462 / 9130 = 70.78%\r\nPath Accuracy (unlabeled, ignorePunc): 6980 / 9130 = 76.45%\r\nParsed 1000 sentences in 20.0 seconds, an average of 49.9 sentences per second.\r\n```\r\n\r\nWith Verbnet + Brown Clusters\r\n\r\n```\r\nUAS: 9215 / 10427 = 88.38%\r\nUAS (no punc): 8114 / 9130 = 88.87%\r\nLAS: 8835 / 10427 = 84.73%\r\nLAS (no punc): 7736 / 9130 = 84.73%\r\nPath Accuracy (labeled, full): 7570 / 10427 = 72.60%\r\nPath Accuracy (unlabeled, full): 8090 / 10427 = 77.59%\r\nPath Accuracy (labeled, ignorePunc): 6510 / 9130 = 71.30%\r\nPath Accuracy (unlabeled, ignorePunc): 7026 / 9130 = 76.96%\r\nParsed 1000 sentences in 24.4 seconds, an average of 41.0 sentences per second.\r\n```\r\nSide note: Training takes a lot longer with the Verbnet features, as recorded in the final timing line.","created_at":"2015-03-17T04:45:03Z","updated_at":"2015-03-17T04:45:03Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/allenai/nlpstack/pulls/109/commits","review_comments_url":"https://api.github.com/repos/allenai/nlpstack/pulls/109/comments","review_comment_url":"https://api.github.com/repos/allenai/nlpstack/pulls/comments{/number}","comments_url":"https://api.github.com/repos/allenai/nlpstack/issues/109/comments","statuses_url":"https://api.github.com/repos/allenai/nlpstack/statuses/e2cc430ed2c46ef3c5b91936fc4bc89544326efa","head":{"label":"sbhaktha:verbnetFeatureChanges","ref":"verbnetFeatureChanges","sha":"e2cc430ed2c46ef3c5b91936fc4bc89544326efa","user":{"login":"sbhaktha","id":5631150,"avatar_url":"https://avatars.githubusercontent.com/u/5631150?v=3","gravatar_id":"","url":"https://api.github.com/users/sbhaktha","html_url":"https://github.com/sbhaktha","followers_url":"https://api.github.com/users/sbhaktha/followers","following_url":"https://api.github.com/users/sbhaktha/following{/other_user}","gists_url":"https://api.github.com/users/sbhaktha/gists{/gist_id}","starred_url":"https://api.github.com/users/sbhaktha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sbhaktha/subscriptions","organizations_url":"https://api.github.com/users/sbhaktha/orgs","repos_url":"https://api.github.com/users/sbhaktha/repos","events_url":"https://api.github.com/users/sbhaktha/events{/privacy}","received_events_url":"https://api.github.com/users/sbhaktha/received_events","type":"User","site_admin":false},"repo":{"id":31919534,"name":"nlpstack","full_name":"sbhaktha/nlpstack","owner":{"login":"sbhaktha","id":5631150,"avatar_url":"https://avatars.githubusercontent.com/u/5631150?v=3","gravatar_id":"","url":"https://api.github.com/users/sbhaktha","html_url":"https://github.com/sbhaktha","followers_url":"https://api.github.com/users/sbhaktha/followers","following_url":"https://api.github.com/users/sbhaktha/following{/other_user}","gists_url":"https://api.github.com/users/sbhaktha/gists{/gist_id}","starred_url":"https://api.github.com/users/sbhaktha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sbhaktha/subscriptions","organizations_url":"https://api.github.com/users/sbhaktha/orgs","repos_url":"https://api.github.com/users/sbhaktha/repos","events_url":"https://api.github.com/users/sbhaktha/events{/privacy}","received_events_url":"https://api.github.com/users/sbhaktha/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/sbhaktha/nlpstack","description":"NLP toolkit (tokenizer, POS-tagger, parser, etc.)","fork":true,"url":"https://api.github.com/repos/sbhaktha/nlpstack","forks_url":"https://api.github.com/repos/sbhaktha/nlpstack/forks","keys_url":"https://api.github.com/repos/sbhaktha/nlpstack/keys{/key_id}","collaborators_url":"https://api.github.com/repos/sbhaktha/nlpstack/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/sbhaktha/nlpstack/teams","hooks_url":"https://api.github.com/repos/sbhaktha/nlpstack/hooks","issue_events_url":"https://api.github.com/repos/sbhaktha/nlpstack/issues/events{/number}","events_url":"https://api.github.com/repos/sbhaktha/nlpstack/events","assignees_url":"https://api.github.com/repos/sbhaktha/nlpstack/assignees{/user}","branches_url":"https://api.github.com/repos/sbhaktha/nlpstack/branches{/branch}","tags_url":"https://api.github.com/repos/sbhaktha/nlpstack/tags","blobs_url":"https://api.github.com/repos/sbhaktha/nlpstack/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/sbhaktha/nlpstack/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/sbhaktha/nlpstack/git/refs{/sha}","trees_url":"https://api.github.com/repos/sbhaktha/nlpstack/git/trees{/sha}","statuses_url":"https://api.github.com/repos/sbhaktha/nlpstack/statuses/{sha}","languages_url":"https://api.github.com/repos/sbhaktha/nlpstack/languages","stargazers_url":"https://api.github.com/repos/sbhaktha/nlpstack/stargazers","contributors_url":"https://api.github.com/repos/sbhaktha/nlpstack/contributors","subscribers_url":"https://api.github.com/repos/sbhaktha/nlpstack/subscribers","subscription_url":"https://api.github.com/repos/sbhaktha/nlpstack/subscription","commits_url":"https://api.github.com/repos/sbhaktha/nlpstack/commits{/sha}","git_commits_url":"https://api.github.com/repos/sbhaktha/nlpstack/git/commits{/sha}","comments_url":"https://api.github.com/repos/sbhaktha/nlpstack/comments{/number}","issue_comment_url":"https://api.github.com/repos/sbhaktha/nlpstack/issues/comments{/number}","contents_url":"https://api.github.com/repos/sbhaktha/nlpstack/contents/{+path}","compare_url":"https://api.github.com/repos/sbhaktha/nlpstack/compare/{base}...{head}","merges_url":"https://api.github.com/repos/sbhaktha/nlpstack/merges","archive_url":"https://api.github.com/repos/sbhaktha/nlpstack/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/sbhaktha/nlpstack/downloads","issues_url":"https://api.github.com/repos/sbhaktha/nlpstack/issues{/number}","pulls_url":"https://api.github.com/repos/sbhaktha/nlpstack/pulls{/number}","milestones_url":"https://api.github.com/repos/sbhaktha/nlpstack/milestones{/number}","notifications_url":"https://api.github.com/repos/sbhaktha/nlpstack/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/sbhaktha/nlpstack/labels{/name}","releases_url":"https://api.github.com/repos/sbhaktha/nlpstack/releases{/id}","created_at":"2015-03-09T19:58:42Z","updated_at":"2015-03-09T19:58:43Z","pushed_at":"2015-03-17T03:14:59Z","git_url":"git://github.com/sbhaktha/nlpstack.git","ssh_url":"git@github.com:sbhaktha/nlpstack.git","clone_url":"https://github.com/sbhaktha/nlpstack.git","svn_url":"https://github.com/sbhaktha/nlpstack","homepage":"http://nlpstack.dev.allenai.org:8062/tools.html","size":1149,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"allenai:master","ref":"master","sha":"1b555aef591b5ef8be7d32555cb0b540cfa0f880","user":{"login":"allenai","id":5667695,"avatar_url":"https://avatars.githubusercontent.com/u/5667695?v=3","gravatar_id":"","url":"https://api.github.com/users/allenai","html_url":"https://github.com/allenai","followers_url":"https://api.github.com/users/allenai/followers","following_url":"https://api.github.com/users/allenai/following{/other_user}","gists_url":"https://api.github.com/users/allenai/gists{/gist_id}","starred_url":"https://api.github.com/users/allenai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/allenai/subscriptions","organizations_url":"https://api.github.com/users/allenai/orgs","repos_url":"https://api.github.com/users/allenai/repos","events_url":"https://api.github.com/users/allenai/events{/privacy}","received_events_url":"https://api.github.com/users/allenai/received_events","type":"Organization","site_admin":false},"repo":{"id":13838096,"name":"nlpstack","full_name":"allenai/nlpstack","owner":{"login":"allenai","id":5667695,"avatar_url":"https://avatars.githubusercontent.com/u/5667695?v=3","gravatar_id":"","url":"https://api.github.com/users/allenai","html_url":"https://github.com/allenai","followers_url":"https://api.github.com/users/allenai/followers","following_url":"https://api.github.com/users/allenai/following{/other_user}","gists_url":"https://api.github.com/users/allenai/gists{/gist_id}","starred_url":"https://api.github.com/users/allenai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/allenai/subscriptions","organizations_url":"https://api.github.com/users/allenai/orgs","repos_url":"https://api.github.com/users/allenai/repos","events_url":"https://api.github.com/users/allenai/events{/privacy}","received_events_url":"https://api.github.com/users/allenai/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/allenai/nlpstack","description":"NLP toolkit (tokenizer, POS-tagger, parser, etc.)","fork":false,"url":"https://api.github.com/repos/allenai/nlpstack","forks_url":"https://api.github.com/repos/allenai/nlpstack/forks","keys_url":"https://api.github.com/repos/allenai/nlpstack/keys{/key_id}","collaborators_url":"https://api.github.com/repos/allenai/nlpstack/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/allenai/nlpstack/teams","hooks_url":"https://api.github.com/repos/allenai/nlpstack/hooks","issue_events_url":"https://api.github.com/repos/allenai/nlpstack/issues/events{/number}","events_url":"https://api.github.com/repos/allenai/nlpstack/events","assignees_url":"https://api.github.com/repos/allenai/nlpstack/assignees{/user}","branches_url":"https://api.github.com/repos/allenai/nlpstack/branches{/branch}","tags_url":"https://api.github.com/repos/allenai/nlpstack/tags","blobs_url":"https://api.github.com/repos/allenai/nlpstack/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/allenai/nlpstack/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/allenai/nlpstack/git/refs{/sha}","trees_url":"https://api.github.com/repos/allenai/nlpstack/git/trees{/sha}","statuses_url":"https://api.github.com/repos/allenai/nlpstack/statuses/{sha}","languages_url":"https://api.github.com/repos/allenai/nlpstack/languages","stargazers_url":"https://api.github.com/repos/allenai/nlpstack/stargazers","contributors_url":"https://api.github.com/repos/allenai/nlpstack/contributors","subscribers_url":"https://api.github.com/repos/allenai/nlpstack/subscribers","subscription_url":"https://api.github.com/repos/allenai/nlpstack/subscription","commits_url":"https://api.github.com/repos/allenai/nlpstack/commits{/sha}","git_commits_url":"https://api.github.com/repos/allenai/nlpstack/git/commits{/sha}","comments_url":"https://api.github.com/repos/allenai/nlpstack/comments{/number}","issue_comment_url":"https://api.github.com/repos/allenai/nlpstack/issues/comments{/number}","contents_url":"https://api.github.com/repos/allenai/nlpstack/contents/{+path}","compare_url":"https://api.github.com/repos/allenai/nlpstack/compare/{base}...{head}","merges_url":"https://api.github.com/repos/allenai/nlpstack/merges","archive_url":"https://api.github.com/repos/allenai/nlpstack/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/allenai/nlpstack/downloads","issues_url":"https://api.github.com/repos/allenai/nlpstack/issues{/number}","pulls_url":"https://api.github.com/repos/allenai/nlpstack/pulls{/number}","milestones_url":"https://api.github.com/repos/allenai/nlpstack/milestones{/number}","notifications_url":"https://api.github.com/repos/allenai/nlpstack/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/allenai/nlpstack/labels{/name}","releases_url":"https://api.github.com/repos/allenai/nlpstack/releases{/id}","created_at":"2013-10-24T16:43:22Z","updated_at":"2015-03-13T21:30:16Z","pushed_at":"2015-03-14T00:29:24Z","git_url":"git://github.com/allenai/nlpstack.git","ssh_url":"git@github.com:allenai/nlpstack.git","clone_url":"https://github.com/allenai/nlpstack.git","svn_url":"https://github.com/allenai/nlpstack","homepage":"http://nlpstack.dev.allenai.org:8062/tools.html","size":2539,"stargazers_count":4,"watchers_count":4,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":19,"forks":2,"open_issues":19,"watchers":4,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/allenai/nlpstack/pulls/109"},"html":{"href":"https://github.com/allenai/nlpstack/pull/109"},"issue":{"href":"https://api.github.com/repos/allenai/nlpstack/issues/109"},"comments":{"href":"https://api.github.com/repos/allenai/nlpstack/issues/109/comments"},"review_comments":{"href":"https://api.github.com/repos/allenai/nlpstack/pulls/109/comments"},"review_comment":{"href":"https://api.github.com/repos/allenai/nlpstack/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/allenai/nlpstack/pulls/109/commits"},"statuses":{"href":"https://api.github.com/repos/allenai/nlpstack/statuses/e2cc430ed2c46ef3c5b91936fc4bc89544326efa"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":2,"additions":238,"deletions":163,"changed_files":14}}