{"action":"created","issue":{"url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/47","labels_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/47/labels{/name}","comments_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/47/comments","events_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/47/events","html_url":"https://github.com/infolab-csail/WikipediaBase/pull/47","id":99378638,"number":47,"title":"Fetch data from wikipedia.org","user":{"login":"alvaromorales","id":2496170,"avatar_url":"https://avatars.githubusercontent.com/u/2496170?v=3","gravatar_id":"","url":"https://api.github.com/users/alvaromorales","html_url":"https://github.com/alvaromorales","followers_url":"https://api.github.com/users/alvaromorales/followers","following_url":"https://api.github.com/users/alvaromorales/following{/other_user}","gists_url":"https://api.github.com/users/alvaromorales/gists{/gist_id}","starred_url":"https://api.github.com/users/alvaromorales/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alvaromorales/subscriptions","organizations_url":"https://api.github.com/users/alvaromorales/orgs","repos_url":"https://api.github.com/users/alvaromorales/repos","events_url":"https://api.github.com/users/alvaromorales/events{/privacy}","received_events_url":"https://api.github.com/users/alvaromorales/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-08-06T07:48:16Z","updated_at":"2015-08-18T17:17:31Z","closed_at":"2015-08-18T17:16:47Z","pull_request":{"url":"https://api.github.com/repos/infolab-csail/WikipediaBase/pulls/47","html_url":"https://github.com/infolab-csail/WikipediaBase/pull/47","diff_url":"https://github.com/infolab-csail/WikipediaBase/pull/47.diff","patch_url":"https://github.com/infolab-csail/WikipediaBase/pull/47.patch"},"body":"Added a fetcher and renderer that fetches data directly from wikipedia.org (see #31). Implemented a CachingFetcher and CachingRenderer as proposed in #46.\r\n\r\nI read the [FAQ](https://www.mediawiki.org/wiki/API:FAQ#get_the_content_of_a_page_.28wikitext.29.3F) on the Wikipedia API and ran experiments to determine the fastest way to get rendered and unrendered pages. I experimented with /index.php, /api.php, and RESTbase, and found that index.php was best for the Fetcher and api.php was best for the renderer.\r\n\r\nAlso fixed #44 -- all strings in WikipediaBase are now unicode, except when outputting an enchantment for START.\r\n\r\nUnfortunately, this ended up being a *huge* change that broke several tests. I was unable to just swap out the implementation of the old WikipediaSiteFetcher because other abstractions like Article or Infobox made assumptions on the format of the data returned by the Fetcher. Wikipedia.org's data was different enough to break some regexes or XPATH expressions.\r\n\r\nApologies for the size of this PR, but I found some bugs along the way that needed to be fixed to get tests to pass.\r\n"},"comment":{"url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/comments/132284458","html_url":"https://github.com/infolab-csail/WikipediaBase/pull/47#issuecomment-132284458","issue_url":"https://api.github.com/repos/infolab-csail/WikipediaBase/issues/47","id":132284458,"user":{"login":"fakedrake","id":408231,"avatar_url":"https://avatars.githubusercontent.com/u/408231?v=3","gravatar_id":"","url":"https://api.github.com/users/fakedrake","html_url":"https://github.com/fakedrake","followers_url":"https://api.github.com/users/fakedrake/followers","following_url":"https://api.github.com/users/fakedrake/following{/other_user}","gists_url":"https://api.github.com/users/fakedrake/gists{/gist_id}","starred_url":"https://api.github.com/users/fakedrake/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fakedrake/subscriptions","organizations_url":"https://api.github.com/users/fakedrake/orgs","repos_url":"https://api.github.com/users/fakedrake/repos","events_url":"https://api.github.com/users/fakedrake/events{/privacy}","received_events_url":"https://api.github.com/users/fakedrake/received_events","type":"User","site_admin":false},"created_at":"2015-08-18T17:17:31Z","updated_at":"2015-08-18T17:17:31Z","body":"Yes sorry, I took a quick look at it as I am not at home. I trust that it's ok :)"}}