{"action":"created","issue":{"url":"https://api.github.com/repos/guilherme-pg/citationhunt/issues/9","labels_url":"https://api.github.com/repos/guilherme-pg/citationhunt/issues/9/labels{/name}","comments_url":"https://api.github.com/repos/guilherme-pg/citationhunt/issues/9/comments","events_url":"https://api.github.com/repos/guilherme-pg/citationhunt/issues/9/events","html_url":"https://github.com/guilherme-pg/citationhunt/issues/9","id":59914541,"number":9,"title":"Alternative approach using live database","user":{"login":"zhaofengli","id":2189609,"avatar_url":"https://avatars.githubusercontent.com/u/2189609?v=3","gravatar_id":"","url":"https://api.github.com/users/zhaofengli","html_url":"https://github.com/zhaofengli","followers_url":"https://api.github.com/users/zhaofengli/followers","following_url":"https://api.github.com/users/zhaofengli/following{/other_user}","gists_url":"https://api.github.com/users/zhaofengli/gists{/gist_id}","starred_url":"https://api.github.com/users/zhaofengli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhaofengli/subscriptions","organizations_url":"https://api.github.com/users/zhaofengli/orgs","repos_url":"https://api.github.com/users/zhaofengli/repos","events_url":"https://api.github.com/users/zhaofengli/events{/privacy}","received_events_url":"https://api.github.com/users/zhaofengli/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-03-05T07:16:15Z","updated_at":"2015-03-06T14:48:23Z","closed_at":null,"body":"In case you are not aware, WMF provides [Tool Labs](https://wikitech.wikimedia.org/wiki/Help:Tool Labs) which is a free hosting service for Wikimedia-related tools, with access to live database replicas and latest dumps. With live database access, I propose another way to discover pages needing citations:\r\n1. Using SQL, select a random page in article space (namespace 0) which transcludes {{cn}} with the `templatelinks` table (see [database layout](https://www.mediawiki.org/wiki/Manual:Database_layout) and [example](http://quarry.wmflabs.org/query/2420))\r\n2. Obtain the page content via API (There doesn't seem to be a way to get this from the database)\r\n3. Parse the page using existing code, to get the uncited part\r\n\r\nThis way, we can avoid the lag caused by the use of dumps (what is shown represents the current state of the article) and there will be no need to pre-process the dumps. Editors also tend to trust tools hosted on Tool Labs more than those on third-party servers.\r\n\r\nTools on Tool Labs need to be licensed under a [OSI-approved license](http://opensource.org/licenses/).\r\n\r\nZhaofeng Li ([User:Zhaofeng Li](https://en.wikipedia.org/wiki/User:Zhaofeng_Li) on Wikipedia)"},"comment":{"url":"https://api.github.com/repos/guilherme-pg/citationhunt/issues/comments/77569919","html_url":"https://github.com/guilherme-pg/citationhunt/issues/9#issuecomment-77569919","issue_url":"https://api.github.com/repos/guilherme-pg/citationhunt/issues/9","id":77569919,"user":{"login":"guilherme-pg","id":489134,"avatar_url":"https://avatars.githubusercontent.com/u/489134?v=3","gravatar_id":"","url":"https://api.github.com/users/guilherme-pg","html_url":"https://github.com/guilherme-pg","followers_url":"https://api.github.com/users/guilherme-pg/followers","following_url":"https://api.github.com/users/guilherme-pg/following{/other_user}","gists_url":"https://api.github.com/users/guilherme-pg/gists{/gist_id}","starred_url":"https://api.github.com/users/guilherme-pg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guilherme-pg/subscriptions","organizations_url":"https://api.github.com/users/guilherme-pg/orgs","repos_url":"https://api.github.com/users/guilherme-pg/repos","events_url":"https://api.github.com/users/guilherme-pg/events{/privacy}","received_events_url":"https://api.github.com/users/guilherme-pg/received_events","type":"User","site_admin":false},"created_at":"2015-03-06T14:48:23Z","updated_at":"2015-03-06T14:48:23Z","body":"Sorry, my comment wasn't clear at all. My concern was that the way we categorize requires all relevant pages and their categories to be known in advance. So even with the live database, some pre-processing will be necessary to figure out which categories will be featured on the site, whereas if we didn't use categories, we could just select a random page and parse it at each access. This is not really a big deal, and is definitely doable, I just wanted to point it out because it was missing from your proposal."}}