{"action":"created","issue":{"url":"https://api.github.com/repos/openaddresses/machine/issues/48","labels_url":"https://api.github.com/repos/openaddresses/machine/issues/48/labels{/name}","comments_url":"https://api.github.com/repos/openaddresses/machine/issues/48/comments","events_url":"https://api.github.com/repos/openaddresses/machine/issues/48/events","html_url":"https://github.com/openaddresses/machine/issues/48","id":54471691,"number":48,"title":"Fix cache type guessing","user":{"login":"migurski","id":58730,"avatar_url":"https://avatars.githubusercontent.com/u/58730?v=3","gravatar_id":"","url":"https://api.github.com/users/migurski","html_url":"https://github.com/migurski","followers_url":"https://api.github.com/users/migurski/followers","following_url":"https://api.github.com/users/migurski/following{/other_user}","gists_url":"https://api.github.com/users/migurski/gists{/gist_id}","starred_url":"https://api.github.com/users/migurski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/migurski/subscriptions","organizations_url":"https://api.github.com/users/migurski/orgs","repos_url":"https://api.github.com/users/migurski/repos","events_url":"https://api.github.com/users/migurski/events{/privacy}","received_events_url":"https://api.github.com/users/migurski/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-01-15T16:40:48Z","updated_at":"2015-01-15T19:15:36Z","closed_at":null,"body":"We have two related problems connected to bad guesses about content-type for cached files. We’re not using the [`compression` core tag](https://github.com/openaddresses/openaddresses/blob/master/CONTRIBUTING.md#core-tags) as a hint, and in many cases [sources mis-report types](https://gist.github.com/migurski/8fe9d98e760795eaa1a7). We’re also not using HTTP status codes to skip processing of 4xx and 5xx responses.\r\n\r\n* [ ] Use `compression` tag to recognize zip files.\r\n* [ ] Use HTTP status responses to recognize missing/bad data.\r\n* [ ] Recover from exceptions inside `process_one.process()`.\r\n\r\nSee sample error log below.\r\n\r\n```\r\n  DEBUG: Content-Type says \"text/html\" for http://stjamesassessor.com/Downloads/DownloadFile.asp?FileID=4206\r\n  DEBUG: Guessed us-la-st_james-72eb71dd.html for http://stjamesassessor.com/Downloads/DownloadFile.asp?FileID=4206\r\n   INFO: Requesting http://stjamesassessor.com/Downloads/DownloadFile.asp?FileID=4206\r\n: Downloaded 158 bytes for file out/process_one-_el6uX/cache-AWFAnY/http/us-la-st_james-72eb71dd.html\r\n   INFO: Cached data in file:///home/ubuntu/out/process_one-_el6uX/cached/us-la-st_james-72eb71dd.html\r\n  DEBUG: URL says \".html\" for file:///home/ubuntu/out/process_one-_el6uX/cached/us-la-st_james-72eb71dd.html\r\n  DEBUG: Guessed us-la-st_james-1f0ec642.html for file:///home/ubuntu/out/process_one-_el6uX/cached/us-la-st_james-72eb71dd.html\r\n  DEBUG: File exists out/process_one-_el6uX/conform-zetfy_/http/us-la-st_james-1f0ec642.html\r\n   INFO: Downloaded to ['out/process_one-_el6uX/conform-zetfy_/http/us-la-st_james-1f0ec642.html']\r\n  ERROR: Error while running process_one.process\r\n\r\nTraceback (most recent call last):\r\n  File \"/usr/local/lib/python2.7/dist-packages/openaddr/jobs.py\", line 61, in _run_process_one\r\n    result = process_one.process(path, destination, extras)\r\n  File \"/usr/local/lib/python2.7/dist-packages/openaddr/process_one.py\", line 38, in process\r\n    conform_result = conform(temp_src, temp_dir, cache_result.todict())\r\n  File \"/usr/local/lib/python2.7/dist-packages/openaddr/__init__.py\", line 149, in conform\r\n    decompressed_paths = task2.decompress(downloaded_path, workdir)\r\n  File \"/usr/local/lib/python2.7/dist-packages/openaddr/conform.py\", line 108, in decompress\r\n    with ZipFile(source_path, 'r') as z:\r\n  File \"/usr/lib/python2.7/zipfile.py\", line 770, in __init__\r\n    self._RealGetContents()\r\n  File \"/usr/lib/python2.7/zipfile.py\", line 811, in _RealGetContents\r\n    raise BadZipfile, \"File is not a zip file\"\r\nBadZipfile: File is not a zip file\r\n```"},"comment":{"url":"https://api.github.com/repos/openaddresses/machine/issues/comments/70143585","html_url":"https://github.com/openaddresses/machine/issues/48#issuecomment-70143585","issue_url":"https://api.github.com/repos/openaddresses/machine/issues/48","id":70143585,"user":{"login":"NelsonMinar","id":382136,"avatar_url":"https://avatars.githubusercontent.com/u/382136?v=3","gravatar_id":"","url":"https://api.github.com/users/NelsonMinar","html_url":"https://github.com/NelsonMinar","followers_url":"https://api.github.com/users/NelsonMinar/followers","following_url":"https://api.github.com/users/NelsonMinar/following{/other_user}","gists_url":"https://api.github.com/users/NelsonMinar/gists{/gist_id}","starred_url":"https://api.github.com/users/NelsonMinar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/NelsonMinar/subscriptions","organizations_url":"https://api.github.com/users/NelsonMinar/orgs","repos_url":"https://api.github.com/users/NelsonMinar/repos","events_url":"https://api.github.com/users/NelsonMinar/events{/privacy}","received_events_url":"https://api.github.com/users/NelsonMinar/received_events","type":"User","site_admin":false},"created_at":"2015-01-15T19:15:36Z","updated_at":"2015-01-15T19:15:36Z","body":"14 files in my last run threw a BadZipfile exception.\r\nca-ab-strathcona-county ca-bc-langley ca-bc-west_kelowna ca-ns-halifax us-ca-san_diego us-fl-collier us-ga-glynn us-la-st_james us-ma us-nc-charlotte us-nc-columbus us-ri us-tx-round_rock us-va-city_of_falls_church"}}