{"action":"created","comment":{"url":"https://api.github.com/repos/meta-toolkit/meta/pulls/comments/38823135","id":38823135,"diff_hunk":"@@ -0,0 +1,44 @@\n+/**\n+ * @file information_gain.cpp\n+ * @author Sean Massung\n+ */\n+\n+#include \"features/information_gain.h\"\n+\n+namespace meta\n+{\n+namespace features\n+{\n+\n+const std::string information_gain::id = \"info-gain\";\n+\n+double information_gain::score(label_id lid, term_id tid) const\n+{\n+    double p_tc = term_and_class(tid, lid);\n+    double p_ntnc = not_term_and_not_class(tid, lid);\n+    double p_ntc = not_term_and_class(tid, lid);\n+    double p_tnc = term_and_not_class(tid, lid);\n+    double p_c = prob_class(lid);\n+    double p_t = prob_term(tid);\n+    double p_nc = 1.0 - p_c;\n+    double p_nt = 1.0 - p_t;\n+\n+    double gain_tc = p_tc * std::log(p_tc / (p_t * p_c));\n+    double gain_ntnc = p_ntnc * std::log(p_ntnc / (p_nt * p_nc));\n+    double gain_ntc = p_ntc * std::log(p_ntc / (p_nt * p_c));\n+    double gain_tnc = p_tnc * std::log(p_tnc / (p_t * p_nc));\n+\n+    // if any denominators were zero, make the expression zero\n+    if (std::isnan(gain_tc))\n+        gain_tc = 0.0;\n+    if (std::isnan(gain_ntnc))\n+        gain_ntnc = 0.0;\n+    if (std::isnan(gain_ntc))\n+        gain_ntc = 0.0;\n+    if (std::isnan(gain_tnc))\n+        gain_tnc = 0.0;","path":"src/features/information_gain.cpp","position":39,"original_position":39,"commit_id":"5c3fc95dca534e9a8581ee050bf07e4b1ed23482","original_commit_id":"64755c2dbc4ac3362a9e09c0606c119cb6bed890","user":{"login":"smassung","id":1396411,"avatar_url":"https://avatars.githubusercontent.com/u/1396411?v=3","gravatar_id":"","url":"https://api.github.com/users/smassung","html_url":"https://github.com/smassung","followers_url":"https://api.github.com/users/smassung/followers","following_url":"https://api.github.com/users/smassung/following{/other_user}","gists_url":"https://api.github.com/users/smassung/gists{/gist_id}","starred_url":"https://api.github.com/users/smassung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smassung/subscriptions","organizations_url":"https://api.github.com/users/smassung/orgs","repos_url":"https://api.github.com/users/smassung/repos","events_url":"https://api.github.com/users/smassung/events{/privacy}","received_events_url":"https://api.github.com/users/smassung/received_events","type":"User","site_admin":false},"body":"Changed to `std::isfinite` which should check for all those cases.","created_at":"2015-09-06T16:29:46Z","updated_at":"2015-09-06T16:29:46Z","html_url":"https://github.com/meta-toolkit/meta/pull/107#discussion_r38823135","pull_request_url":"https://api.github.com/repos/meta-toolkit/meta/pulls/107","_links":{"self":{"href":"https://api.github.com/repos/meta-toolkit/meta/pulls/comments/38823135"},"html":{"href":"https://github.com/meta-toolkit/meta/pull/107#discussion_r38823135"},"pull_request":{"href":"https://api.github.com/repos/meta-toolkit/meta/pulls/107"}}},"pull_request":{"url":"https://api.github.com/repos/meta-toolkit/meta/pulls/107","id":44273476,"html_url":"https://github.com/meta-toolkit/meta/pull/107","diff_url":"https://github.com/meta-toolkit/meta/pull/107.diff","patch_url":"https://github.com/meta-toolkit/meta/pull/107.patch","issue_url":"https://api.github.com/repos/meta-toolkit/meta/issues/107","number":107,"state":"open","locked":false,"title":"Merge language-model branch","user":{"login":"smassung","id":1396411,"avatar_url":"https://avatars.githubusercontent.com/u/1396411?v=3","gravatar_id":"","url":"https://api.github.com/users/smassung","html_url":"https://github.com/smassung","followers_url":"https://api.github.com/users/smassung/followers","following_url":"https://api.github.com/users/smassung/following{/other_user}","gists_url":"https://api.github.com/users/smassung/gists{/gist_id}","starred_url":"https://api.github.com/users/smassung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smassung/subscriptions","organizations_url":"https://api.github.com/users/smassung/orgs","repos_url":"https://api.github.com/users/smassung/repos","events_url":"https://api.github.com/users/smassung/events{/privacy}","received_events_url":"https://api.github.com/users/smassung/received_events","type":"User","site_admin":false},"body":"The `language-model` branch has become a `sean's-features-branch`, and I think it's in an acceptable state to merge before it gets too far off from `develop`. It would also be great to include features from `language-model` in MeTA 2.0 when `develop` itself is finally merged.\r\n\r\nOne group of additions are the `language_model`, `sentence`,  `diff`, and `diff_analyzer` classes. These all complement one another. While perfectly useable on their own, there are two things we should look into as (I suggest) separate features:\r\n\r\n- potentially unify `sentence` and `sequence` somehow; there is a decent amount of overlap, though I can't decide whether they should be separate classes or not\r\n- language model estimation; currently the LM requires an `.arpa` file (at least before it creates its own binary files)\r\n- an efficient language model `top_k` function; this is used to predict the top *k* most likely tokens given a context. The way the current LM is built is optimized for ngram query speed. While possible to run the `top_k` function, it requires iterating over the unigram vocabulary. In order to solve this, some additional (most likely tree-based) data structures will probably have to be added\r\n\r\nThe other group of additions are the `feature_selector`s. These are based on the old feature selectors, except they operate on `disk_vector`s and tokenized/analyzed documents. A demo app called `feature-summary` shows one way how they can be used. Again, this is perfectly useable as-is, but would be better with:\r\n\r\n- creating `selected_index`, or whatever you want to call it. It's another wrapped index object that uses the feature selectors' `selected(term_id)` function to check whether to search for (and cache) terms. Theoretically this should be simple enough to write, but I just haven't gotten around to it yet.\r\n\r\nIn summary, there are many new features, with four additional \"wanted\" features. I'd like to get the current code into develop and work on the wanted features as separate issues. @skystrife , let's see what you think.","created_at":"2015-09-05T00:59:58Z","updated_at":"2015-09-06T16:29:46Z","closed_at":null,"merged_at":null,"merge_commit_sha":"784a405fd011a4c000f800c6f0a5ca118669bf4f","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/meta-toolkit/meta/pulls/107/commits","review_comments_url":"https://api.github.com/repos/meta-toolkit/meta/pulls/107/comments","review_comment_url":"https://api.github.com/repos/meta-toolkit/meta/pulls/comments{/number}","comments_url":"https://api.github.com/repos/meta-toolkit/meta/issues/107/comments","statuses_url":"https://api.github.com/repos/meta-toolkit/meta/statuses/5c3fc95dca534e9a8581ee050bf07e4b1ed23482","head":{"label":"meta-toolkit:language-model","ref":"language-model","sha":"5c3fc95dca534e9a8581ee050bf07e4b1ed23482","user":{"login":"meta-toolkit","id":6588029,"avatar_url":"https://avatars.githubusercontent.com/u/6588029?v=3","gravatar_id":"","url":"https://api.github.com/users/meta-toolkit","html_url":"https://github.com/meta-toolkit","followers_url":"https://api.github.com/users/meta-toolkit/followers","following_url":"https://api.github.com/users/meta-toolkit/following{/other_user}","gists_url":"https://api.github.com/users/meta-toolkit/gists{/gist_id}","starred_url":"https://api.github.com/users/meta-toolkit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/meta-toolkit/subscriptions","organizations_url":"https://api.github.com/users/meta-toolkit/orgs","repos_url":"https://api.github.com/users/meta-toolkit/repos","events_url":"https://api.github.com/users/meta-toolkit/events{/privacy}","received_events_url":"https://api.github.com/users/meta-toolkit/received_events","type":"Organization","site_admin":false},"repo":{"id":16466317,"name":"meta","full_name":"meta-toolkit/meta","owner":{"login":"meta-toolkit","id":6588029,"avatar_url":"https://avatars.githubusercontent.com/u/6588029?v=3","gravatar_id":"","url":"https://api.github.com/users/meta-toolkit","html_url":"https://github.com/meta-toolkit","followers_url":"https://api.github.com/users/meta-toolkit/followers","following_url":"https://api.github.com/users/meta-toolkit/following{/other_user}","gists_url":"https://api.github.com/users/meta-toolkit/gists{/gist_id}","starred_url":"https://api.github.com/users/meta-toolkit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/meta-toolkit/subscriptions","organizations_url":"https://api.github.com/users/meta-toolkit/orgs","repos_url":"https://api.github.com/users/meta-toolkit/repos","events_url":"https://api.github.com/users/meta-toolkit/events{/privacy}","received_events_url":"https://api.github.com/users/meta-toolkit/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/meta-toolkit/meta","description":"A Modern C++ Data Sciences Toolkit","fork":false,"url":"https://api.github.com/repos/meta-toolkit/meta","forks_url":"https://api.github.com/repos/meta-toolkit/meta/forks","keys_url":"https://api.github.com/repos/meta-toolkit/meta/keys{/key_id}","collaborators_url":"https://api.github.com/repos/meta-toolkit/meta/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/meta-toolkit/meta/teams","hooks_url":"https://api.github.com/repos/meta-toolkit/meta/hooks","issue_events_url":"https://api.github.com/repos/meta-toolkit/meta/issues/events{/number}","events_url":"https://api.github.com/repos/meta-toolkit/meta/events","assignees_url":"https://api.github.com/repos/meta-toolkit/meta/assignees{/user}","branches_url":"https://api.github.com/repos/meta-toolkit/meta/branches{/branch}","tags_url":"https://api.github.com/repos/meta-toolkit/meta/tags","blobs_url":"https://api.github.com/repos/meta-toolkit/meta/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/meta-toolkit/meta/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/meta-toolkit/meta/git/refs{/sha}","trees_url":"https://api.github.com/repos/meta-toolkit/meta/git/trees{/sha}","statuses_url":"https://api.github.com/repos/meta-toolkit/meta/statuses/{sha}","languages_url":"https://api.github.com/repos/meta-toolkit/meta/languages","stargazers_url":"https://api.github.com/repos/meta-toolkit/meta/stargazers","contributors_url":"https://api.github.com/repos/meta-toolkit/meta/contributors","subscribers_url":"https://api.github.com/repos/meta-toolkit/meta/subscribers","subscription_url":"https://api.github.com/repos/meta-toolkit/meta/subscription","commits_url":"https://api.github.com/repos/meta-toolkit/meta/commits{/sha}","git_commits_url":"https://api.github.com/repos/meta-toolkit/meta/git/commits{/sha}","comments_url":"https://api.github.com/repos/meta-toolkit/meta/comments{/number}","issue_comment_url":"https://api.github.com/repos/meta-toolkit/meta/issues/comments{/number}","contents_url":"https://api.github.com/repos/meta-toolkit/meta/contents/{+path}","compare_url":"https://api.github.com/repos/meta-toolkit/meta/compare/{base}...{head}","merges_url":"https://api.github.com/repos/meta-toolkit/meta/merges","archive_url":"https://api.github.com/repos/meta-toolkit/meta/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/meta-toolkit/meta/downloads","issues_url":"https://api.github.com/repos/meta-toolkit/meta/issues{/number}","pulls_url":"https://api.github.com/repos/meta-toolkit/meta/pulls{/number}","milestones_url":"https://api.github.com/repos/meta-toolkit/meta/milestones{/number}","notifications_url":"https://api.github.com/repos/meta-toolkit/meta/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/meta-toolkit/meta/labels{/name}","releases_url":"https://api.github.com/repos/meta-toolkit/meta/releases{/id}","created_at":"2014-02-02T23:54:34Z","updated_at":"2015-09-04T04:01:59Z","pushed_at":"2015-09-06T15:32:50Z","git_url":"git://github.com/meta-toolkit/meta.git","ssh_url":"git@github.com:meta-toolkit/meta.git","clone_url":"https://github.com/meta-toolkit/meta.git","svn_url":"https://github.com/meta-toolkit/meta","homepage":"https://meta-toolkit.org","size":32359,"stargazers_count":191,"watchers_count":191,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":82,"mirror_url":null,"open_issues_count":18,"forks":82,"open_issues":18,"watchers":191,"default_branch":"master"}},"base":{"label":"meta-toolkit:develop","ref":"develop","sha":"e44c0811118874ef6a4172543f49dfcbf57a6fe9","user":{"login":"meta-toolkit","id":6588029,"avatar_url":"https://avatars.githubusercontent.com/u/6588029?v=3","gravatar_id":"","url":"https://api.github.com/users/meta-toolkit","html_url":"https://github.com/meta-toolkit","followers_url":"https://api.github.com/users/meta-toolkit/followers","following_url":"https://api.github.com/users/meta-toolkit/following{/other_user}","gists_url":"https://api.github.com/users/meta-toolkit/gists{/gist_id}","starred_url":"https://api.github.com/users/meta-toolkit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/meta-toolkit/subscriptions","organizations_url":"https://api.github.com/users/meta-toolkit/orgs","repos_url":"https://api.github.com/users/meta-toolkit/repos","events_url":"https://api.github.com/users/meta-toolkit/events{/privacy}","received_events_url":"https://api.github.com/users/meta-toolkit/received_events","type":"Organization","site_admin":false},"repo":{"id":16466317,"name":"meta","full_name":"meta-toolkit/meta","owner":{"login":"meta-toolkit","id":6588029,"avatar_url":"https://avatars.githubusercontent.com/u/6588029?v=3","gravatar_id":"","url":"https://api.github.com/users/meta-toolkit","html_url":"https://github.com/meta-toolkit","followers_url":"https://api.github.com/users/meta-toolkit/followers","following_url":"https://api.github.com/users/meta-toolkit/following{/other_user}","gists_url":"https://api.github.com/users/meta-toolkit/gists{/gist_id}","starred_url":"https://api.github.com/users/meta-toolkit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/meta-toolkit/subscriptions","organizations_url":"https://api.github.com/users/meta-toolkit/orgs","repos_url":"https://api.github.com/users/meta-toolkit/repos","events_url":"https://api.github.com/users/meta-toolkit/events{/privacy}","received_events_url":"https://api.github.com/users/meta-toolkit/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/meta-toolkit/meta","description":"A Modern C++ Data Sciences Toolkit","fork":false,"url":"https://api.github.com/repos/meta-toolkit/meta","forks_url":"https://api.github.com/repos/meta-toolkit/meta/forks","keys_url":"https://api.github.com/repos/meta-toolkit/meta/keys{/key_id}","collaborators_url":"https://api.github.com/repos/meta-toolkit/meta/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/meta-toolkit/meta/teams","hooks_url":"https://api.github.com/repos/meta-toolkit/meta/hooks","issue_events_url":"https://api.github.com/repos/meta-toolkit/meta/issues/events{/number}","events_url":"https://api.github.com/repos/meta-toolkit/meta/events","assignees_url":"https://api.github.com/repos/meta-toolkit/meta/assignees{/user}","branches_url":"https://api.github.com/repos/meta-toolkit/meta/branches{/branch}","tags_url":"https://api.github.com/repos/meta-toolkit/meta/tags","blobs_url":"https://api.github.com/repos/meta-toolkit/meta/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/meta-toolkit/meta/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/meta-toolkit/meta/git/refs{/sha}","trees_url":"https://api.github.com/repos/meta-toolkit/meta/git/trees{/sha}","statuses_url":"https://api.github.com/repos/meta-toolkit/meta/statuses/{sha}","languages_url":"https://api.github.com/repos/meta-toolkit/meta/languages","stargazers_url":"https://api.github.com/repos/meta-toolkit/meta/stargazers","contributors_url":"https://api.github.com/repos/meta-toolkit/meta/contributors","subscribers_url":"https://api.github.com/repos/meta-toolkit/meta/subscribers","subscription_url":"https://api.github.com/repos/meta-toolkit/meta/subscription","commits_url":"https://api.github.com/repos/meta-toolkit/meta/commits{/sha}","git_commits_url":"https://api.github.com/repos/meta-toolkit/meta/git/commits{/sha}","comments_url":"https://api.github.com/repos/meta-toolkit/meta/comments{/number}","issue_comment_url":"https://api.github.com/repos/meta-toolkit/meta/issues/comments{/number}","contents_url":"https://api.github.com/repos/meta-toolkit/meta/contents/{+path}","compare_url":"https://api.github.com/repos/meta-toolkit/meta/compare/{base}...{head}","merges_url":"https://api.github.com/repos/meta-toolkit/meta/merges","archive_url":"https://api.github.com/repos/meta-toolkit/meta/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/meta-toolkit/meta/downloads","issues_url":"https://api.github.com/repos/meta-toolkit/meta/issues{/number}","pulls_url":"https://api.github.com/repos/meta-toolkit/meta/pulls{/number}","milestones_url":"https://api.github.com/repos/meta-toolkit/meta/milestones{/number}","notifications_url":"https://api.github.com/repos/meta-toolkit/meta/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/meta-toolkit/meta/labels{/name}","releases_url":"https://api.github.com/repos/meta-toolkit/meta/releases{/id}","created_at":"2014-02-02T23:54:34Z","updated_at":"2015-09-04T04:01:59Z","pushed_at":"2015-09-06T15:32:50Z","git_url":"git://github.com/meta-toolkit/meta.git","ssh_url":"git@github.com:meta-toolkit/meta.git","clone_url":"https://github.com/meta-toolkit/meta.git","svn_url":"https://github.com/meta-toolkit/meta","homepage":"https://meta-toolkit.org","size":32359,"stargazers_count":191,"watchers_count":191,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":82,"mirror_url":null,"open_issues_count":18,"forks":82,"open_issues":18,"watchers":191,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/meta-toolkit/meta/pulls/107"},"html":{"href":"https://github.com/meta-toolkit/meta/pull/107"},"issue":{"href":"https://api.github.com/repos/meta-toolkit/meta/issues/107"},"comments":{"href":"https://api.github.com/repos/meta-toolkit/meta/issues/107/comments"},"review_comments":{"href":"https://api.github.com/repos/meta-toolkit/meta/pulls/107/comments"},"review_comment":{"href":"https://api.github.com/repos/meta-toolkit/meta/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/meta-toolkit/meta/pulls/107/commits"},"statuses":{"href":"https://api.github.com/repos/meta-toolkit/meta/statuses/5c3fc95dca534e9a8581ee050bf07e4b1ed23482"}}}}