{"action":"created","comment":{"url":"https://api.github.com/repos/infolab-csail/WikipediaBase/pulls/comments/47437137","id":47437137,"diff_hunk":"@@ -36,49 +42,89 @@ def __call__(self, symbol, *av, **kw):\n \n class TermClassifier(BaseClassifier):\n \n-    def classify(self, symbol, fetcher=None):\n+    def classify(self, symbol):\n         return ['wikibase-term']\n \n \n class SectionsClassifier(BaseClassifier):\n \n-    def classify(self, symbol, fetcher=None):\n+    def classify(self, symbol):\n         return ['wikibase-sections']\n \n \n class InfoboxClassifier(BaseClassifier):\n \n-    def classify(self, symbol, fetcher=None):\n+    def classify(self, symbol):\n         classes = []\n-        for ibox in get_infoboxes(symbol, fetcher=fetcher):\n-            classes.append(ibox.wikipedia_class())\n+        for ibox in get_infoboxes(symbol):\n+            classes.append(ibox.cls)\n         return classes\n \n \n class PersonClassifier(BaseClassifier):\n \n+    CATEGORY_REGEXES = [re.compile(p) for p in [\n+        \".* people\",\n+        \".* person\",\n+        \"^\\d+ deaths.*\",\n+        \"^\\d+ births.*\",\n+        \".* actors\",\n+        \".* musicians\",\n+        \".* players\",\n+        \".* singers\",\n+    ]]\n+\n+    CATEGORY_MATCHES = [","path":"wikipediabase/classifiers.py","position":69,"original_position":69,"commit_id":"84b15458357924af9e793b4205423f75fdc80be9","original_commit_id":"9de98ea83b4e119360efc02b81c535c274815cae","user":{"login":"alvaromorales","id":2496170,"avatar_url":"https://avatars.githubusercontent.com/u/2496170?v=3","gravatar_id":"","url":"https://api.github.com/users/alvaromorales","html_url":"https://github.com/alvaromorales","followers_url":"https://api.github.com/users/alvaromorales/followers","following_url":"https://api.github.com/users/alvaromorales/following{/other_user}","gists_url":"https://api.github.com/users/alvaromorales/gists{/gist_id}","starred_url":"https://api.github.com/users/alvaromorales/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alvaromorales/subscriptions","organizations_url":"https://api.github.com/users/alvaromorales/orgs","repos_url":"https://api.github.com/users/alvaromorales/repos","events_url":"https://api.github.com/users/alvaromorales/events{/privacy}","received_events_url":"https://api.github.com/users/alvaromorales/received_events","type":"User","site_admin":false},"body":"These \"categories\" are actually Wikipedia article categories.\r\n\r\nThis method of classifying `wikibase-person` works surprisingly well. It was translated from Ruby-Wikipediabase. These regexes capture the most common categories assigned to people articles.","created_at":"2015-12-12T21:37:38Z","updated_at":"2015-12-12T21:37:38Z","html_url":"https://github.com/infolab-csail/WikipediaBase/pull/99#discussion_r47437137","pull_request_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/pulls/99","_links":{"self":{"href":"https://api.github.com/repos/infolab-csail/WikipediaBase/pulls/comments/47437137"},"html":{"href":"https://github.com/infolab-csail/WikipediaBase/pull/99#discussion_r47437137"},"pull_request":{"href":"https://api.github.com/repos/infolab-csail/WikipediaBase/pulls/99"}}},"pull_request":{"url":"https://api.github.com/repos/infolab-csail/WikipediaBase/pulls/99","id":52070046,"html_url":"https://github.com/infolab-csail/WikipediaBase/pull/99","diff_url":"https://github.com/infolab-csail/WikipediaBase/pull/99.diff","patch_url":"https://github.com/infolab-csail/WikipediaBase/pull/99.patch","issue_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/99","number":99,"state":"open","locked":false,"title":"Improve WikipediaBase performance by adding a backend","user":{"login":"alvaromorales","id":2496170,"avatar_url":"https://avatars.githubusercontent.com/u/2496170?v=3","gravatar_id":"","url":"https://api.github.com/users/alvaromorales","html_url":"https://github.com/alvaromorales","followers_url":"https://api.github.com/users/alvaromorales/followers","following_url":"https://api.github.com/users/alvaromorales/following{/other_user}","gists_url":"https://api.github.com/users/alvaromorales/gists{/gist_id}","starred_url":"https://api.github.com/users/alvaromorales/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alvaromorales/subscriptions","organizations_url":"https://api.github.com/users/alvaromorales/orgs","repos_url":"https://api.github.com/users/alvaromorales/repos","events_url":"https://api.github.com/users/alvaromorales/events{/privacy}","received_events_url":"https://api.github.com/users/alvaromorales/received_events","type":"User","site_admin":false},"body":"This PR addresses the performance issues of WikipediaBase.\r\n\r\nAdded scripts to build the WikipediaBase backend. The backend has one table `articles` with three columns: `id, title, markup`. It uses the [peewee](http://docs.peewee-orm.com/en/latest/) ORM to abstract low-level SQL calls. Peewee is lightweight and the setup was trivial. It made things significantly easier than SQLAlchemy.\r\n\r\nA main design goal with the backend was to make it as simple as possible. The simpler it is, the easier to rebuild and keep up to date. We're using Postgres as a backend, in an effort to use one database system for InfoLab services.\r\n\r\nAdded scripts to generate symbols and synonyms. The bulk of this work was translating from Ruby code. Fixed the bug with \"bad\" symbols that @michaelsilver documented in Ruby-WikipediaBase by recursively following redirects and disambiguations.\r\n\r\nBuilding the backend from a Wikipedia dump takes about 10 hours. Generating symbols and synonyms takes 40 hours.\r\n\r\nThe main performance improvement was obtained by not having to fetch HTML for every request. We only need rendered HTML for `get`; we use the markup in the DB for `get-classes`, `get-attributes`, `sort-symbols`, and `sort-symbols-named`. Benchmarking shows that **WikipediaBase is at least as fast as Ruby-WikipediaBase** for most requests. In order to do this, we split infobox construction into two steps: `Infobox` only uses markup, and `RenderedInfobox` uses markup and HTML.\r\n\r\nDid some cleanup of the code by removing circular dependencies between modules. Added symbols and synonyms files using Git Large File Service for Github. We're now going to be able to diff these files every time we rebuild the backend.\r\n\r\nWe're still going to be able to use Travis for CI. Instead of reading from a database, we default to fetch markup live from wikipedia.org when tests run on the Travis platform.\r\n\r\nApologies for the large PR -- adding a backend changed the code in many places. The good news is that Python-WikipediaBase is now running on kiribati, with a fresh backend and new symbols. We're starting a production test. We are on track for our goal to have WikipediaBase running in production by December.\r\n\r\nFixes #85 \r\nFixes #70 \r\nFixes #93 ","created_at":"2015-11-29T20:24:22Z","updated_at":"2015-12-12T21:37:38Z","closed_at":null,"merged_at":null,"merge_commit_sha":"075451e73b489ae2ed2019c855825bb07bc998f8","assignee":{"login":"michaelsilver","id":5913522,"avatar_url":"https://avatars.githubusercontent.com/u/5913522?v=3","gravatar_id":"","url":"https://api.github.com/users/michaelsilver","html_url":"https://github.com/michaelsilver","followers_url":"https://api.github.com/users/michaelsilver/followers","following_url":"https://api.github.com/users/michaelsilver/following{/other_user}","gists_url":"https://api.github.com/users/michaelsilver/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelsilver/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelsilver/subscriptions","organizations_url":"https://api.github.com/users/michaelsilver/orgs","repos_url":"https://api.github.com/users/michaelsilver/repos","events_url":"https://api.github.com/users/michaelsilver/events{/privacy}","received_events_url":"https://api.github.com/users/michaelsilver/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/pulls/99/commits","review_comments_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/pulls/99/comments","review_comment_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/pulls/comments{/number}","comments_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/99/comments","statuses_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/statuses/84b15458357924af9e793b4205423f75fdc80be9","head":{"label":"infolab-csail:backend","ref":"backend","sha":"84b15458357924af9e793b4205423f75fdc80be9","user":{"login":"infolab-csail","id":13588310,"avatar_url":"https://avatars.githubusercontent.com/u/13588310?v=3","gravatar_id":"","url":"https://api.github.com/users/infolab-csail","html_url":"https://github.com/infolab-csail","followers_url":"https://api.github.com/users/infolab-csail/followers","following_url":"https://api.github.com/users/infolab-csail/following{/other_user}","gists_url":"https://api.github.com/users/infolab-csail/gists{/gist_id}","starred_url":"https://api.github.com/users/infolab-csail/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/infolab-csail/subscriptions","organizations_url":"https://api.github.com/users/infolab-csail/orgs","repos_url":"https://api.github.com/users/infolab-csail/repos","events_url":"https://api.github.com/users/infolab-csail/events{/privacy}","received_events_url":"https://api.github.com/users/infolab-csail/received_events","type":"Organization","site_admin":false},"repo":{"id":15544139,"name":"WikipediaBase","full_name":"infolab-csail/WikipediaBase","owner":{"login":"infolab-csail","id":13588310,"avatar_url":"https://avatars.githubusercontent.com/u/13588310?v=3","gravatar_id":"","url":"https://api.github.com/users/infolab-csail","html_url":"https://github.com/infolab-csail","followers_url":"https://api.github.com/users/infolab-csail/followers","following_url":"https://api.github.com/users/infolab-csail/following{/other_user}","gists_url":"https://api.github.com/users/infolab-csail/gists{/gist_id}","starred_url":"https://api.github.com/users/infolab-csail/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/infolab-csail/subscriptions","organizations_url":"https://api.github.com/users/infolab-csail/orgs","repos_url":"https://api.github.com/users/infolab-csail/repos","events_url":"https://api.github.com/users/infolab-csail/events{/privacy}","received_events_url":"https://api.github.com/users/infolab-csail/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/infolab-csail/WikipediaBase","description":"Wikipedia interaction with start.mit.edu","fork":false,"url":"https://api.github.com/repos/infolab-csail/WikipediaBase","forks_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/forks","keys_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/keys{/key_id}","collaborators_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/teams","hooks_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/hooks","issue_events_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/events{/number}","events_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/events","assignees_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/assignees{/user}","branches_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/branches{/branch}","tags_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/tags","blobs_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/git/refs{/sha}","trees_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/git/trees{/sha}","statuses_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/statuses/{sha}","languages_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/languages","stargazers_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/stargazers","contributors_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/contributors","subscribers_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/subscribers","subscription_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/subscription","commits_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/commits{/sha}","git_commits_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/git/commits{/sha}","comments_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/comments{/number}","issue_comment_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/comments{/number}","contents_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/contents/{+path}","compare_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/compare/{base}...{head}","merges_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/merges","archive_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/downloads","issues_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues{/number}","pulls_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/pulls{/number}","milestones_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/milestones{/number}","notifications_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/labels{/name}","releases_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/releases{/id}","created_at":"2013-12-31T06:02:34Z","updated_at":"2015-08-11T08:24:18Z","pushed_at":"2015-12-12T21:32:24Z","git_url":"git://github.com/infolab-csail/WikipediaBase.git","ssh_url":"git@github.com:infolab-csail/WikipediaBase.git","clone_url":"https://github.com/infolab-csail/WikipediaBase.git","svn_url":"https://github.com/infolab-csail/WikipediaBase","homepage":null,"size":8549,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":36,"forks":2,"open_issues":36,"watchers":0,"default_branch":"master"}},"base":{"label":"infolab-csail:master","ref":"master","sha":"ab5aa92786bddcd7942ad3e3f1f4e433575ba3fb","user":{"login":"infolab-csail","id":13588310,"avatar_url":"https://avatars.githubusercontent.com/u/13588310?v=3","gravatar_id":"","url":"https://api.github.com/users/infolab-csail","html_url":"https://github.com/infolab-csail","followers_url":"https://api.github.com/users/infolab-csail/followers","following_url":"https://api.github.com/users/infolab-csail/following{/other_user}","gists_url":"https://api.github.com/users/infolab-csail/gists{/gist_id}","starred_url":"https://api.github.com/users/infolab-csail/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/infolab-csail/subscriptions","organizations_url":"https://api.github.com/users/infolab-csail/orgs","repos_url":"https://api.github.com/users/infolab-csail/repos","events_url":"https://api.github.com/users/infolab-csail/events{/privacy}","received_events_url":"https://api.github.com/users/infolab-csail/received_events","type":"Organization","site_admin":false},"repo":{"id":15544139,"name":"WikipediaBase","full_name":"infolab-csail/WikipediaBase","owner":{"login":"infolab-csail","id":13588310,"avatar_url":"https://avatars.githubusercontent.com/u/13588310?v=3","gravatar_id":"","url":"https://api.github.com/users/infolab-csail","html_url":"https://github.com/infolab-csail","followers_url":"https://api.github.com/users/infolab-csail/followers","following_url":"https://api.github.com/users/infolab-csail/following{/other_user}","gists_url":"https://api.github.com/users/infolab-csail/gists{/gist_id}","starred_url":"https://api.github.com/users/infolab-csail/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/infolab-csail/subscriptions","organizations_url":"https://api.github.com/users/infolab-csail/orgs","repos_url":"https://api.github.com/users/infolab-csail/repos","events_url":"https://api.github.com/users/infolab-csail/events{/privacy}","received_events_url":"https://api.github.com/users/infolab-csail/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/infolab-csail/WikipediaBase","description":"Wikipedia interaction with start.mit.edu","fork":false,"url":"https://api.github.com/repos/infolab-csail/WikipediaBase","forks_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/forks","keys_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/keys{/key_id}","collaborators_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/teams","hooks_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/hooks","issue_events_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/events{/number}","events_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/events","assignees_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/assignees{/user}","branches_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/branches{/branch}","tags_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/tags","blobs_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/git/refs{/sha}","trees_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/git/trees{/sha}","statuses_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/statuses/{sha}","languages_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/languages","stargazers_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/stargazers","contributors_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/contributors","subscribers_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/subscribers","subscription_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/subscription","commits_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/commits{/sha}","git_commits_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/git/commits{/sha}","comments_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/comments{/number}","issue_comment_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/comments{/number}","contents_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/contents/{+path}","compare_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/compare/{base}...{head}","merges_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/merges","archive_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/downloads","issues_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues{/number}","pulls_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/pulls{/number}","milestones_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/milestones{/number}","notifications_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/labels{/name}","releases_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/releases{/id}","created_at":"2013-12-31T06:02:34Z","updated_at":"2015-08-11T08:24:18Z","pushed_at":"2015-12-12T21:32:24Z","git_url":"git://github.com/infolab-csail/WikipediaBase.git","ssh_url":"git@github.com:infolab-csail/WikipediaBase.git","clone_url":"https://github.com/infolab-csail/WikipediaBase.git","svn_url":"https://github.com/infolab-csail/WikipediaBase","homepage":null,"size":8549,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":36,"forks":2,"open_issues":36,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/infolab-csail/WikipediaBase/pulls/99"},"html":{"href":"https://github.com/infolab-csail/WikipediaBase/pull/99"},"issue":{"href":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/99"},"comments":{"href":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/99/comments"},"review_comments":{"href":"https://api.github.com/repos/infolab-csail/WikipediaBase/pulls/99/comments"},"review_comment":{"href":"https://api.github.com/repos/infolab-csail/WikipediaBase/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/infolab-csail/WikipediaBase/pulls/99/commits"},"statuses":{"href":"https://api.github.com/repos/infolab-csail/WikipediaBase/statuses/84b15458357924af9e793b4205423f75fdc80be9"}}}}