{"action":"created","issue":{"url":"https://api.github.com/repos/mdn/browsercompat/issues/36","labels_url":"https://api.github.com/repos/mdn/browsercompat/issues/36/labels{/name}","comments_url":"https://api.github.com/repos/mdn/browsercompat/issues/36/comments","events_url":"https://api.github.com/repos/mdn/browsercompat/issues/36/events","html_url":"https://github.com/mdn/browsercompat/pull/36","id":99677111,"number":36,"title":"fix bug 1182542 - Scrape deeply nested MDN pages, other fixes","user":{"login":"jwhitlock","id":286017,"avatar_url":"https://avatars.githubusercontent.com/u/286017?v=3","gravatar_id":"","url":"https://api.github.com/users/jwhitlock","html_url":"https://github.com/jwhitlock","followers_url":"https://api.github.com/users/jwhitlock/followers","following_url":"https://api.github.com/users/jwhitlock/following{/other_user}","gists_url":"https://api.github.com/users/jwhitlock/gists{/gist_id}","starred_url":"https://api.github.com/users/jwhitlock/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jwhitlock/subscriptions","organizations_url":"https://api.github.com/users/jwhitlock/orgs","repos_url":"https://api.github.com/users/jwhitlock/repos","events_url":"https://api.github.com/users/jwhitlock/events{/privacy}","received_events_url":"https://api.github.com/users/jwhitlock/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"groovecoder","id":71928,"avatar_url":"https://avatars.githubusercontent.com/u/71928?v=3","gravatar_id":"","url":"https://api.github.com/users/groovecoder","html_url":"https://github.com/groovecoder","followers_url":"https://api.github.com/users/groovecoder/followers","following_url":"https://api.github.com/users/groovecoder/following{/other_user}","gists_url":"https://api.github.com/users/groovecoder/gists{/gist_id}","starred_url":"https://api.github.com/users/groovecoder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/groovecoder/subscriptions","organizations_url":"https://api.github.com/users/groovecoder/orgs","repos_url":"https://api.github.com/users/groovecoder/repos","events_url":"https://api.github.com/users/groovecoder/events{/privacy}","received_events_url":"https://api.github.com/users/groovecoder/received_events","type":"User","site_admin":false},"milestone":null,"comments":2,"created_at":"2015-08-07T15:55:21Z","updated_at":"2015-08-10T20:22:48Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/mdn/browsercompat/pulls/36","html_url":"https://github.com/mdn/browsercompat/pull/36","diff_url":"https://github.com/mdn/browsercompat/pull/36.diff","patch_url":"https://github.com/mdn/browsercompat/pull/36.patch"},"body":"Florian Sholtz and the MDN team have reduced a lot of the [importer issues](https://browsercompat.herokuapp.com/importer/issues), and have requested a re-scrape of MDN to see if new content has additional issues.  This PR includes tool improvements that will help the process.  This code has **not** been run against https://browsercompat.herokuapp.com, since we're trying to do code reviews before \"production\" pushes.\r\n\r\n* Fix [bug 1182542](https://bugzilla.mozilla.org/show_bug.cgi?id=1182542) - Use recursive calls to ``$children`` to discover MDN URLs more than 5 levels deep.\r\n* Handle various error conditions with ``$children`` API\r\n* Make it easy to get fresh content with ``--no-cache``\r\n* Re-scrape MDN with ``import_mdn.py``.\r\n\r\nIf you want to run this locally:\r\n* Setup browsercompat project\r\n  * Add a superuser with / username+password\r\n  * Optionally populate with data from https://github.com/mdn/browsercompat-data (will have less \"unknown_version\" type errors)\r\n  * Optionally run with memcache \r\n* With a good network connection and power supply, run:\r\n  *  ``time tools/mirror_mdn_features.py`` - after about 60 minutes, will prompt to make changes, then 5 - 10 minutes to commit changes.  For me, got 841 new pages, 19 changed, 969 deleted, 6023 the same.\r\n  * ``time tools/import_mdn.py`` - takes about 6.5 hours to parse 5877 pages.\r\n"},"comment":{"url":"https://api.github.com/repos/mdn/browsercompat/issues/comments/129595365","html_url":"https://github.com/mdn/browsercompat/pull/36#issuecomment-129595365","issue_url":"https://api.github.com/repos/mdn/browsercompat/issues/36","id":129595365,"user":{"login":"groovecoder","id":71928,"avatar_url":"https://avatars.githubusercontent.com/u/71928?v=3","gravatar_id":"","url":"https://api.github.com/users/groovecoder","html_url":"https://github.com/groovecoder","followers_url":"https://api.github.com/users/groovecoder/followers","following_url":"https://api.github.com/users/groovecoder/following{/other_user}","gists_url":"https://api.github.com/users/groovecoder/gists{/gist_id}","starred_url":"https://api.github.com/users/groovecoder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/groovecoder/subscriptions","organizations_url":"https://api.github.com/users/groovecoder/orgs","repos_url":"https://api.github.com/users/groovecoder/repos","events_url":"https://api.github.com/users/groovecoder/events{/privacy}","received_events_url":"https://api.github.com/users/groovecoder/received_events","type":"User","site_admin":false},"created_at":"2015-08-10T20:22:48Z","updated_at":"2015-08-10T20:22:48Z","body":"Code looks good. When I tried to run:\r\n```\r\n./tools/upload_data.py --data /Users/lcrouch/code/browsercompat-data/data\r\n```\r\nI got:\r\n```\r\nTraceback (most recent call last):\r\n  File \"./tools/upload_data.py\", line 62, in <module>\r\n    changes = tool.run()\r\n  File \"./tools/upload_data.py\", line 39, in run\r\n    return self.sync_changes(api_collection, local_collection)\r\n  File \"/Users/lcrouch/code/browsercompat/tools/common.py\", line 311, in sync_changes\r\n    return changeset.change_original_collection()\r\n  File \"/Users/lcrouch/code/browsercompat/tools/resources.py\", line 673, in change_original_collection\r\n    resource_type, json_api[resource_type])\r\n  File \"/Users/lcrouch/code/browsercompat/tools/client.py\", line 147, in create\r\n    response = self.request('POST', resource_type, data=data)\r\n  File \"/Users/lcrouch/code/browsercompat/tools/client.py\", line 83, in request\r\n    response.status_code, response.content)\r\nclient.APIException: (u'POST http://localhost:8000/api/v1/versions: Unexpected response', 400, '{\\n    \"errors\": [\\n        {\\n            \"status\": \"400\",\\n            \"detail\": \"This field may not be blank.\",\\n            \"path\": \"/version\"\\n        }\\n    ]\\n}')\r\n```"}}