{"action":"created","issue":{"url":"https://api.github.com/repos/ipfs/archives/issues/28","labels_url":"https://api.github.com/repos/ipfs/archives/issues/28/labels{/name}","comments_url":"https://api.github.com/repos/ipfs/archives/issues/28/comments","events_url":"https://api.github.com/repos/ipfs/archives/issues/28/events","html_url":"https://github.com/ipfs/archives/issues/28","id":108431815,"number":28,"title":"IPFS as a backend to a web archiving","user":{"login":"ikreymer","id":1015759,"avatar_url":"https://avatars.githubusercontent.com/u/1015759?v=3","gravatar_id":"","url":"https://api.github.com/users/ikreymer","html_url":"https://github.com/ikreymer","followers_url":"https://api.github.com/users/ikreymer/followers","following_url":"https://api.github.com/users/ikreymer/following{/other_user}","gists_url":"https://api.github.com/users/ikreymer/gists{/gist_id}","starred_url":"https://api.github.com/users/ikreymer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ikreymer/subscriptions","organizations_url":"https://api.github.com/users/ikreymer/orgs","repos_url":"https://api.github.com/users/ikreymer/repos","events_url":"https://api.github.com/users/ikreymer/events{/privacy}","received_events_url":"https://api.github.com/users/ikreymer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-09-26T01:20:57Z","updated_at":"2015-09-27T00:07:52Z","closed_at":null,"body":"I am building a new on-demand web archiving system, called webrecorder.io, which allows for on-demand archiving of any web site (by acting as a rewriting + recording proxy).\r\nThis version (actually beta.webrecorder.io) will soon be open-sourced and will be available for users to deploy on their own.\r\n\r\nThe system allows for a user to create recording of any web site, including dynamic content, by browsing it through a the recorder, eg. https://webrecorder.io/record/example.com/ and replay by browsing through replay, https://webrecorder.io/replay/example.com/\r\n\r\nThe recording is a [WARC](https://en.wikipedia.org/wiki/Web_ARChive) file, a standard used by Internet Archive and other archiving orgs. The file can be broken down into records (basically contents of HTTP response + request and extra metadata), and each of these records could be put individually into IPFS.\r\n\r\nI suppose this sort of relates to ipfs/archives#7 but perhaps in a more sophisticated way.\r\n\r\nMost obvious mode of operation: Store each WARC record in IPFS individually.\r\n\r\nSome unknowns (to me):\r\n  - Resolving URL + datetime to the hash of the stored object in IPFS (This is also part of the memento protocol). Basically looking up a url and datetime and mapping it to an IPFS hash.\r\n\r\n  - Privacy / security concerns: Would want to have users create private archives, or be able to set controls on what is accessible to whom. This is not specific to web archiving, but something I don't (yet) know much about.\r\n \r\n\r\nFor more reference:\r\nThe system is built using these tools: https://github.com/ikreymer/pywb , https://github.com/ikreymer/warcprox\r\nAn older simplified version of the \"webrecorder\" concept: https://github.com/ikreymer/pywb-webrecorder."},"comment":{"url":"https://api.github.com/repos/ipfs/archives/issues/comments/143506387","html_url":"https://github.com/ipfs/archives/issues/28#issuecomment-143506387","issue_url":"https://api.github.com/repos/ipfs/archives/issues/28","id":143506387,"user":{"login":"ikreymer","id":1015759,"avatar_url":"https://avatars.githubusercontent.com/u/1015759?v=3","gravatar_id":"","url":"https://api.github.com/users/ikreymer","html_url":"https://github.com/ikreymer","followers_url":"https://api.github.com/users/ikreymer/followers","following_url":"https://api.github.com/users/ikreymer/following{/other_user}","gists_url":"https://api.github.com/users/ikreymer/gists{/gist_id}","starred_url":"https://api.github.com/users/ikreymer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ikreymer/subscriptions","organizations_url":"https://api.github.com/users/ikreymer/orgs","repos_url":"https://api.github.com/users/ikreymer/repos","events_url":"https://api.github.com/users/ikreymer/events{/privacy}","received_events_url":"https://api.github.com/users/ikreymer/received_events","type":"User","site_admin":false},"created_at":"2015-09-27T00:07:52Z","updated_at":"2015-09-27T00:07:52Z","body":"Well, taking a step back, I realized that the directory structure is immutable, so perhaps this directory structure idea isn't as useful as I had thought.. I had assumed ti could be used as updatable index, but of course that's not the case..\r\n\r\nHmm.. I think perhaps key-value store or simple database ipfs/ipfs#82 would be useful for querying and update the index.. \r\n\r\nThe directory structure isn't really needed, as it would just indicate a particular set of files in a WARC, or the order in which something was recorded, which is arbitrary.. and not the total archive.\r\n\r\nCan the URL and datetime can be embedded as file system metadata stored with the file data? Is that possible?\r\n\r\nWhat is needed is some sort of updatable index... Still trying to understand how IPFS works, sorry :)\r\n\r\n\r\n> How does verification work?\r\n\r\nOh I should probably update that.. It just signs the WARC using https://github.com/ikreymer/warcsigner\r\nso that it can be verified that the WARC was created with Webrecorder and not 'tampered with'. Pretty basic.\r\n\r\n\r\n"}}