{"action":"opened","issue":{"url":"https://api.github.com/repos/attardi/wikiextractor/issues/11","labels_url":"https://api.github.com/repos/attardi/wikiextractor/issues/11/labels{/name}","comments_url":"https://api.github.com/repos/attardi/wikiextractor/issues/11/comments","events_url":"https://api.github.com/repos/attardi/wikiextractor/issues/11/events","html_url":"https://github.com/attardi/wikiextractor/issues/11","id":70355827,"number":11,"title":"UnicodeDecodeError when parsing Chinese Wikipedia","user":{"login":"jmaxx","id":9569570,"avatar_url":"https://avatars.githubusercontent.com/u/9569570?v=3","gravatar_id":"","url":"https://api.github.com/users/jmaxx","html_url":"https://github.com/jmaxx","followers_url":"https://api.github.com/users/jmaxx/followers","following_url":"https://api.github.com/users/jmaxx/following{/other_user}","gists_url":"https://api.github.com/users/jmaxx/gists{/gist_id}","starred_url":"https://api.github.com/users/jmaxx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmaxx/subscriptions","organizations_url":"https://api.github.com/users/jmaxx/orgs","repos_url":"https://api.github.com/users/jmaxx/repos","events_url":"https://api.github.com/users/jmaxx/events{/privacy}","received_events_url":"https://api.github.com/users/jmaxx/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-04-23T09:16:57Z","updated_at":"2015-04-23T09:16:57Z","closed_at":null,"body":"On parsing the Chinese Wikipedia (https://dumps.wikimedia.org/zhwiki/20150325/), I get the following error:\r\n\r\n```\r\nINFO: 596814\t肖卡特·阿齐兹\r\nException in thread Thread-1:\r\nTraceback (most recent call last):\r\n  File \"/home/jan/anaconda/lib/python2.7/threading.py\", line 810, in __bootstrap_inner\r\n    self.run()\r\n  File \"WikiExtractor.py\", line 2290, in run\r\n    job.extract(self._splitter)\r\n  File \"WikiExtractor.py\", line 399, in extract\r\n    text = clean(self, text)\r\n  File \"WikiExtractor.py\", line 1821, in clean\r\n    text = extractor.expandTemplates(text)\r\n  File \"WikiExtractor.py\", line 450, in expandTemplates\r\n    res += wikitext[cur:s] + self.expandTemplate(wikitext[s+2:e-2])\r\n  File \"WikiExtractor.py\", line 671, in expandTemplate\r\n    value = self.expandTemplates(instantiated)\r\n  File \"WikiExtractor.py\", line 450, in expandTemplates\r\n    res += wikitext[cur:s] + self.expandTemplate(wikitext[s+2:e-2])\r\n  File \"WikiExtractor.py\", line 671, in expandTemplate\r\n    value = self.expandTemplates(instantiated)\r\n  File \"WikiExtractor.py\", line 450, in expandTemplates\r\n    res += wikitext[cur:s] + self.expandTemplate(wikitext[s+2:e-2])\r\n  File \"WikiExtractor.py\", line 606, in expandTemplate\r\n    return self.expandTemplates(ret)\r\n  File \"WikiExtractor.py\", line 450, in expandTemplates\r\n    res += wikitext[cur:s] + self.expandTemplate(wikitext[s+2:e-2])\r\n  File \"WikiExtractor.py\", line 606, in expandTemplate\r\n    return self.expandTemplates(ret)\r\n  File \"WikiExtractor.py\", line 450, in expandTemplates\r\n    res += wikitext[cur:s] + self.expandTemplate(wikitext[s+2:e-2])\r\n  File \"WikiExtractor.py\", line 606, in expandTemplate\r\n    return self.expandTemplates(ret)\r\n  File \"WikiExtractor.py\", line 450, in expandTemplates\r\n    res += wikitext[cur:s] + self.expandTemplate(wikitext[s+2:e-2])\r\n  File \"WikiExtractor.py\", line 671, in expandTemplate\r\n    value = self.expandTemplates(instantiated)\r\n  File \"WikiExtractor.py\", line 450, in expandTemplates\r\n    res += wikitext[cur:s] + self.expandTemplate(wikitext[s+2:e-2])\r\n  File \"WikiExtractor.py\", line 582, in expandTemplate\r\n    title = self.expandTemplates(parts[0].strip())\r\n  File \"WikiExtractor.py\", line 450, in expandTemplates\r\n    res += wikitext[cur:s] + self.expandTemplate(wikitext[s+2:e-2])\r\nUnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 2: ordinal not in range(128)\r\n```\r\n\r\nThe line causing the error does not explicitly invoke any ASCII decoding mechanism so I have no idea why it throws an error, but I guess there is a Chinese (or another non ASCII) character in the template. This StackOverflow question (http://stackoverflow.com/questions/21393758/unicodedecodeerror-ascii-codec-cant-decode-byte-0xe5-in-position-0-ordinal) suggest to add\r\n```python\r\nimport sys\r\nreload(sys)\r\nsys.setdefaultencoding(\"utf-8\")\r\n```\r\nat the top of the document. I added the last two lines after the import lines at the top of the script and let the script run over it again (using the previously extracted templates) and it works. I also tried it without the reload command but that didn't work and I even found an explanation for that issue: http://stackoverflow.com/questions/3828723/why-we-need-sys-setdefaultencodingutf-8-in-a-py-script\r\nSo I would suggest to add \r\n```python\r\nreload(sys)\r\nsys.setdefaultencoding(\"utf-8\")\r\n```\r\nto the script as it shouldn't hurt the extraction process (I can file a pull request if you like).\r\nThanks for your work! :)"}}