{"action":"closed","issue":{"url":"https://api.github.com/repos/t3nsor/quora-backup/issues/1","labels_url":"https://api.github.com/repos/t3nsor/quora-backup/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/t3nsor/quora-backup/issues/1/comments","events_url":"https://api.github.com/repos/t3nsor/quora-backup/issues/1/events","html_url":"https://github.com/t3nsor/quora-backup/issues/1","id":59941688,"number":1,"title":"Crawler fails for people with non-ascii names","user":{"login":"eivindorama","id":1726243,"avatar_url":"https://avatars.githubusercontent.com/u/1726243?v=3","gravatar_id":"","url":"https://api.github.com/users/eivindorama","html_url":"https://github.com/eivindorama","followers_url":"https://api.github.com/users/eivindorama/followers","following_url":"https://api.github.com/users/eivindorama/following{/other_user}","gists_url":"https://api.github.com/users/eivindorama/gists{/gist_id}","starred_url":"https://api.github.com/users/eivindorama/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eivindorama/subscriptions","organizations_url":"https://api.github.com/users/eivindorama/orgs","repos_url":"https://api.github.com/users/eivindorama/repos","events_url":"https://api.github.com/users/eivindorama/events{/privacy}","received_events_url":"https://api.github.com/users/eivindorama/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-03-05T11:53:38Z","updated_at":"2015-03-06T04:26:58Z","closed_at":"2015-03-06T04:26:58Z","body":"When trying to run the crawler as directed, I get the following error:\r\n\r\n[DEBUG] Loading input file content.json\r\nTraceback (most recent call last):\r\n  File \"./crawler.py\", line 97, in <module>\r\n    answers = json.load(input_file)\r\n  File \"/usr/lib/python3.4/json/__init__.py\", line 268, in load\r\n    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)\r\n  File \"/usr/lib/python3.4/json/__init__.py\", line 318, in loads\r\n    return _default_decoder.decode(s)\r\n  File \"/usr/lib/python3.4/json/decoder.py\", line 343, in decode\r\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\r\n  File \"/usr/lib/python3.4/json/decoder.py\", line 359, in raw_decode\r\n    obj, end = self.scan_once(s, idx)\r\nValueError: Invalid control character at: line 1 column 133 (char 132)\r\n\r\nLooking at the actual json-file it seems likely to me that the problem here is that I'm registered on Quora as \"Eivind Kjørstad\", and the ø gets encoded in the urls in a way the crawler does not approve of. Specifically the start of my json-file looks like this:\r\n\r\n[[\"https://www.quora.com/What-can-I-do-to-prevent-myself-from-getting-into-a-cycle-of-mediocrity/answer/Eivind-Kj%C3%B8rstad\",\"Added\r\n 1h \r\nago\"],[\"https://www.quora.com/Laws-in-India/If-I-get-a-signed-document-from-my-wife-and-her-parents-saying-that-No-dowry-has-been-given-to-the-groom-and-his-family-in-this-marriage-and-other-necessary-details-can-my-wife-or-her-family-members-still-file-a-case-against-me-under-the-Dowry-Act/answer/Eivind-Kj%C3%B8rstad\",\"Added\r\n 2h \r\nago\"],[\"https://www.quora.com/Why-do-so-many-men-on-Quora-use-the-word-females-to-refer-to-human-women/answer/Eivind-Kj%C3%B8rstad\",\"Added\r\n Wed\"]\r\n\r\nAt a guess, the crawler disapprove of the \"%C3%B8\" part in my name. If I run sed on the json to take out those characters, then the crawler runs, but of course then the URLs it tries to fetch are incorrect and nothing is fetched."}}