{"action":"created","issue":{"url":"https://api.github.com/repos/HTTPArchive/httparchive/issues/6","labels_url":"https://api.github.com/repos/HTTPArchive/httparchive/issues/6/labels{/name}","comments_url":"https://api.github.com/repos/HTTPArchive/httparchive/issues/6/comments","events_url":"https://api.github.com/repos/HTTPArchive/httparchive/issues/6/events","html_url":"https://github.com/HTTPArchive/httparchive/issues/6","id":25740767,"number":6,"title":"Store response bodies","user":{"login":"yoavweiss","id":786187,"avatar_url":"https://avatars.githubusercontent.com/u/786187?v=3","gravatar_id":"","url":"https://api.github.com/users/yoavweiss","html_url":"https://github.com/yoavweiss","followers_url":"https://api.github.com/users/yoavweiss/followers","following_url":"https://api.github.com/users/yoavweiss/following{/other_user}","gists_url":"https://api.github.com/users/yoavweiss/gists{/gist_id}","starred_url":"https://api.github.com/users/yoavweiss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yoavweiss/subscriptions","organizations_url":"https://api.github.com/users/yoavweiss/orgs","repos_url":"https://api.github.com/users/yoavweiss/repos","events_url":"https://api.github.com/users/yoavweiss/events{/privacy}","received_events_url":"https://api.github.com/users/yoavweiss/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2014-01-16T17:14:13Z","updated_at":"2015-12-03T23:07:16Z","closed_at":"2014-06-13T21:09:25Z","body":"It'd be great if we could save the response bodies alongside the meta data, so that they'd be available for download, and possibly for querying later on. That would obsolete slightly overlapping efforts like [webdevdata.org](http://webdevdata.org/), and would enable that group's members (e.g. me :) ) to help out with the HTTPArchive.\r\n\r\nWe should probably keep the actual response bodies off the database, and just save them to disk, while only keeping the file name in the database.\r\n\r\nSince we're talking about a lot of data, it probably makes sense to start out with text based resources only, and see about images later on. \r\n\r\nThoughts?"},"comment":{"url":"https://api.github.com/repos/HTTPArchive/httparchive/issues/comments/161817887","html_url":"https://github.com/HTTPArchive/httparchive/issues/6#issuecomment-161817887","issue_url":"https://api.github.com/repos/HTTPArchive/httparchive/issues/6","id":161817887,"user":{"login":"pmeenan","id":444165,"avatar_url":"https://avatars.githubusercontent.com/u/444165?v=3","gravatar_id":"","url":"https://api.github.com/users/pmeenan","html_url":"https://github.com/pmeenan","followers_url":"https://api.github.com/users/pmeenan/followers","following_url":"https://api.github.com/users/pmeenan/following{/other_user}","gists_url":"https://api.github.com/users/pmeenan/gists{/gist_id}","starred_url":"https://api.github.com/users/pmeenan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pmeenan/subscriptions","organizations_url":"https://api.github.com/users/pmeenan/orgs","repos_url":"https://api.github.com/users/pmeenan/repos","events_url":"https://api.github.com/users/pmeenan/events{/privacy}","received_events_url":"https://api.github.com/users/pmeenan/received_events","type":"User","site_admin":false},"created_at":"2015-12-03T23:07:16Z","updated_at":"2015-12-03T23:07:16Z","body":"We are storing them in a public bucket in Google cloud storage (and will also be making it available in big query). You'll need a developer account to rsync the folders but it is free. I'll have some UI with the bucket paths shortly. The 11/15 crawl was the first one and all 4 crawls uploaded correctly and every crawl going forward should also. \n\n\n\n\n\n\nOn Thu, Dec 3, 2015 at 2:22 PM -0800, \"h3nce\" <notifications@github.com> wrote:\n\n\n\n\n\n\n\n\n\n\n\n\nHi @pmeenan, \n\n\n\n.zip files are working up to February 2015 (http://httparchive.webpagetest.org/habodies.php?run=20150215). Any chance of a refresh for Nov-Dec 2015?\n\n\n\nThanks,\n\n\n\nTom\n\n\n\n—\nReply to this email directly or view it on GitHub.\n\n\n  \n  \n\n\n\n\n\n\n\n\n"}}