{"action":"created","issue":{"url":"https://api.github.com/repos/iipc/webarchive-commons/issues/41","labels_url":"https://api.github.com/repos/iipc/webarchive-commons/issues/41/labels{/name}","comments_url":"https://api.github.com/repos/iipc/webarchive-commons/issues/41/comments","events_url":"https://api.github.com/repos/iipc/webarchive-commons/issues/41/events","html_url":"https://github.com/iipc/webarchive-commons/issues/41","id":58070946,"number":41,"title":"Error-prone HTTP-header parsing in ARCRecord","user":{"login":"tokee","id":199454,"avatar_url":"https://avatars.githubusercontent.com/u/199454?v=3","gravatar_id":"","url":"https://api.github.com/users/tokee","html_url":"https://github.com/tokee","followers_url":"https://api.github.com/users/tokee/followers","following_url":"https://api.github.com/users/tokee/following{/other_user}","gists_url":"https://api.github.com/users/tokee/gists{/gist_id}","starred_url":"https://api.github.com/users/tokee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tokee/subscriptions","organizations_url":"https://api.github.com/users/tokee/orgs","repos_url":"https://api.github.com/users/tokee/repos","events_url":"https://api.github.com/users/tokee/events{/privacy}","received_events_url":"https://api.github.com/users/tokee/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-02-18T14:01:06Z","updated_at":"2015-02-18T20:33:36Z","closed_at":null,"body":"ARCRecord encapsulates streaming of record content, hardening against parsing mistakes. Unfortunately HTTP-headers are processed on the raw ARC-stream, allowing parsing of problematic headers to stream into other records, as demonstrated in issue #40. This results in aborted iteration of ARC records.\r\n\r\nA more conservative approach would be to wrap the record block (HTTP-headers + content) as soon as possible; right after the ARC-entry-header-line has been parsed. Extraction of HTTP-headers would then take place within the hardened stream, isolating any parsing or data problems to the current record."},"comment":{"url":"https://api.github.com/repos/iipc/webarchive-commons/issues/comments/74942254","html_url":"https://github.com/iipc/webarchive-commons/issues/41#issuecomment-74942254","issue_url":"https://api.github.com/repos/iipc/webarchive-commons/issues/41","id":74942254,"user":{"login":"tokee","id":199454,"avatar_url":"https://avatars.githubusercontent.com/u/199454?v=3","gravatar_id":"","url":"https://api.github.com/users/tokee","html_url":"https://github.com/tokee","followers_url":"https://api.github.com/users/tokee/followers","following_url":"https://api.github.com/users/tokee/following{/other_user}","gists_url":"https://api.github.com/users/tokee/gists{/gist_id}","starred_url":"https://api.github.com/users/tokee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tokee/subscriptions","organizations_url":"https://api.github.com/users/tokee/orgs","repos_url":"https://api.github.com/users/tokee/repos","events_url":"https://api.github.com/users/tokee/events{/privacy}","received_events_url":"https://api.github.com/users/tokee/received_events","type":"User","site_admin":false},"created_at":"2015-02-18T20:33:36Z","updated_at":"2015-02-18T20:33:36Z","body":"We tested 95 ARC-files with known problem against the code in issue #40 and a single one failed at the record:\r\n```\r\nhttp://192.168.16.2/robots.txt 192.168.16.2 20111211155955 text/html 111\r\nHTTP/1.1 404 Not found\r\nServer: eHTTP v2.0\r\nConnection: Keep-Alive\r\nContent-Type: text/html\r\nContent-Length:  10\r\n```\r\nThe lines above were immediately followed by the header of the next record. So no content, although the HTTP-header promised 10 bytes. However, the size of the full record (the 5 HTTP-header-lines) were exactly 111 bytes, as stated in the record header. So I guess the ARC-record is technically correct, and the server just returned 0 bytes as content?\r\n\r\nWithout having checked properly, I would guess that the readHttpParser reads through the 5 HTTP-headers and into the following record as there are no terminating blank line to signal the end of the HTTPheaders. Guarding against that would imply a limit on the bytes read in the readHttpParser and then we're back to the whole stream hardening.\r\n\r\nI could take a stab at this, but I am afraid that it won't be a trivial patch as there would have to be multiple changes to avoid double-wrapping."}}