{"action":"created","issue":{"url":"https://api.github.com/repos/ipfs/archives/issues/28","labels_url":"https://api.github.com/repos/ipfs/archives/issues/28/labels{/name}","comments_url":"https://api.github.com/repos/ipfs/archives/issues/28/comments","events_url":"https://api.github.com/repos/ipfs/archives/issues/28/events","html_url":"https://github.com/ipfs/archives/issues/28","id":108431815,"number":28,"title":"IPFS as a backend to a web archiving","user":{"login":"ikreymer","id":1015759,"avatar_url":"https://avatars.githubusercontent.com/u/1015759?v=3","gravatar_id":"","url":"https://api.github.com/users/ikreymer","html_url":"https://github.com/ikreymer","followers_url":"https://api.github.com/users/ikreymer/followers","following_url":"https://api.github.com/users/ikreymer/following{/other_user}","gists_url":"https://api.github.com/users/ikreymer/gists{/gist_id}","starred_url":"https://api.github.com/users/ikreymer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ikreymer/subscriptions","organizations_url":"https://api.github.com/users/ikreymer/orgs","repos_url":"https://api.github.com/users/ikreymer/repos","events_url":"https://api.github.com/users/ikreymer/events{/privacy}","received_events_url":"https://api.github.com/users/ikreymer/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ipfs/archives/labels/spec","name":"spec","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":18,"created_at":"2015-09-26T01:20:57Z","updated_at":"2015-10-18T06:39:16Z","closed_at":null,"body":"I am building a new on-demand web archiving system, called webrecorder.io, which allows for on-demand archiving of any web site (by acting as a rewriting + recording proxy).\r\nThis version (actually beta.webrecorder.io) will soon be open-sourced and will be available for users to deploy on their own.\r\n\r\nThe system allows for a user to create recording of any web site, including dynamic content, by browsing it through a the recorder, eg. https://webrecorder.io/record/example.com/ and replay by browsing through replay, https://webrecorder.io/replay/example.com/\r\n\r\nThe recording is a [WARC](https://en.wikipedia.org/wiki/Web_ARChive) file, a standard used by Internet Archive and other archiving orgs. The file can be broken down into records (basically contents of HTTP response + request and extra metadata), and each of these records could be put individually into IPFS.\r\n\r\nI suppose this sort of relates to ipfs/archives#7 but perhaps in a more sophisticated way.\r\n\r\nMost obvious mode of operation: Store each WARC record in IPFS individually.\r\n\r\nSome unknowns (to me):\r\n  - Resolving URL + datetime to the hash of the stored object in IPFS (This is also part of the memento protocol). Basically looking up a url and datetime and mapping it to an IPFS hash.\r\n\r\n  - Privacy / security concerns: Would want to have users create private archives, or be able to set controls on what is accessible to whom. This is not specific to web archiving, but something I don't (yet) know much about.\r\n \r\n\r\nFor more reference:\r\nThe system is built using these tools: https://github.com/ikreymer/pywb , https://github.com/ikreymer/warcprox\r\nAn older simplified version of the \"webrecorder\" concept: https://github.com/ikreymer/pywb-webrecorder."},"comment":{"url":"https://api.github.com/repos/ipfs/archives/issues/comments/148985268","html_url":"https://github.com/ipfs/archives/issues/28#issuecomment-148985268","issue_url":"https://api.github.com/repos/ipfs/archives/issues/28","id":148985268,"user":{"login":"ikreymer","id":1015759,"avatar_url":"https://avatars.githubusercontent.com/u/1015759?v=3","gravatar_id":"","url":"https://api.github.com/users/ikreymer","html_url":"https://github.com/ikreymer","followers_url":"https://api.github.com/users/ikreymer/followers","following_url":"https://api.github.com/users/ikreymer/following{/other_user}","gists_url":"https://api.github.com/users/ikreymer/gists{/gist_id}","starred_url":"https://api.github.com/users/ikreymer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ikreymer/subscriptions","organizations_url":"https://api.github.com/users/ikreymer/orgs","repos_url":"https://api.github.com/users/ikreymer/repos","events_url":"https://api.github.com/users/ikreymer/events{/privacy}","received_events_url":"https://api.github.com/users/ikreymer/received_events","type":"User","site_admin":false},"created_at":"2015-10-18T06:39:16Z","updated_at":"2015-10-18T06:39:16Z","body":"Thanks everyone for your help. Sorry for delay, I have been busy releasing a new version of webrecorder.io, now out.. Also, happy to announce that webrecorder is now fully open source, at https://github.com/webrecorder/webrecorder\r\n\r\nThere's still a lot to be done before integration would be possible, but I am planning to work on a separate prototype write and replaying WARCs for now..\r\n\r\nAlso, look forward to stopping by the Tuesday meetup in SF and meeting folks in person..\r\n"}}