{"action":"created","issue":{"url":"https://api.github.com/repos/pouchdb/pouchdb/issues/3961","labels_url":"https://api.github.com/repos/pouchdb/pouchdb/issues/3961/labels{/name}","comments_url":"https://api.github.com/repos/pouchdb/pouchdb/issues/3961/comments","events_url":"https://api.github.com/repos/pouchdb/pouchdb/issues/3961/events","html_url":"https://github.com/pouchdb/pouchdb/issues/3961","id":87942569,"number":3961,"title":"Attachments loss","user":{"login":"ermouth","id":1540047,"avatar_url":"https://avatars.githubusercontent.com/u/1540047?v=3","gravatar_id":"","url":"https://api.github.com/users/ermouth","html_url":"https://github.com/ermouth","followers_url":"https://api.github.com/users/ermouth/followers","following_url":"https://api.github.com/users/ermouth/following{/other_user}","gists_url":"https://api.github.com/users/ermouth/gists{/gist_id}","starred_url":"https://api.github.com/users/ermouth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ermouth/subscriptions","organizations_url":"https://api.github.com/users/ermouth/orgs","repos_url":"https://api.github.com/users/ermouth/repos","events_url":"https://api.github.com/users/ermouth/events{/privacy}","received_events_url":"https://api.github.com/users/ermouth/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-06-13T06:08:28Z","updated_at":"2015-06-14T08:47:31Z","closed_at":null,"body":"On https://github.com/pouchdb/pouchdb/issues/3932, https://github.com/pouchdb/pouchdb/issues/3942.\r\n\r\nUPD. Disregard all below, both fix and reason seem to be invalid :(\r\n\r\nSeems that issue takes place in, at least, 3.2.1 and 3.3.1 and, possibly, 3.6.0 (did not test) because:\r\n* timeout in ajax module is too short (1e4 hardcoded, smth around 1e6 look more reasonable)\r\n* remote DB can have several docs with large atts (tens of megabytes) and fetching them induces timeout\r\n* this case is somehow incorrectly handled\r\n* doc is saved with original md5, but zero-length attachment(s).\r\n\r\nIf all above is correct it seems that Pouch never checks if attachment received from remote source is consistent to its digest.\r\n\r\nAs for 3.2.1 there was enough to increase timeout. For general fix I think it‘s reasonable to re-hash received data and revisit timeout handler.\r\n\r\nAlso this case shows why browser tests with CouchDB on `localhost` better be avoided. Virtual loop in general has much wider bandwidth capacity than any real end-user network.\r\n\r\nI can reproduce it in all browsers with remote DB that contains several docs with large attachments. Please let me know if you also can. No code example since case is very specific to network conditions and DB, sorry.\r\n\r\nSeems that https://github.com/pouchdb/pouchdb/issues/3932 could not be reproduced, even repeating steps provided exactly, because my remote DB had several large docs and my network is nearly always very busy.\r\n\r\nTo note – general practice to avoid this kind of issues is to have a test for each numeric constant inside the code, as well as for assumption under each particular constant."},"comment":{"url":"https://api.github.com/repos/pouchdb/pouchdb/issues/comments/111800669","html_url":"https://github.com/pouchdb/pouchdb/issues/3961#issuecomment-111800669","issue_url":"https://api.github.com/repos/pouchdb/pouchdb/issues/3961","id":111800669,"user":{"login":"ermouth","id":1540047,"avatar_url":"https://avatars.githubusercontent.com/u/1540047?v=3","gravatar_id":"","url":"https://api.github.com/users/ermouth","html_url":"https://github.com/ermouth","followers_url":"https://api.github.com/users/ermouth/followers","following_url":"https://api.github.com/users/ermouth/following{/other_user}","gists_url":"https://api.github.com/users/ermouth/gists{/gist_id}","starred_url":"https://api.github.com/users/ermouth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ermouth/subscriptions","organizations_url":"https://api.github.com/users/ermouth/orgs","repos_url":"https://api.github.com/users/ermouth/repos","events_url":"https://api.github.com/users/ermouth/events{/privacy}","received_events_url":"https://api.github.com/users/ermouth/received_events","type":"User","site_admin":false},"created_at":"2015-06-14T08:47:31Z","updated_at":"2015-06-14T08:47:31Z","body":"All below is for Chrome.\r\n\r\nI went down to IDB level and what I see is that Blobs can be fetched well with direct transactions, they have correct type, correct size, but I can not read these Blobs using FileReader. Any available method (readAsArrayBuffer, asBinaryString, asDataUrl, asText) – all fail with error \"NotFound\".\r\n\r\nCode, that fetches attach, looks like this (think it‘s correct):\r\n\r\n``` javascript\r\nvar res, req = indexedDB.open('_pouch_dbname');\r\nreq.onsuccess = function(e) {\r\n  var db = e.target.result;\r\n  db.transaction([\"attach-store\"], \"readonly\")\r\n  .objectStore(\"attach-store\")\r\n  .get(\"md5-SBwxEsOQ1mwFXc3DLe4YSQ==\")\r\n  .onsuccess = function(e) {\r\n  \tconsole.log(res = e.target.result)\t\r\n  }\r\n};\r\n```\r\n`res.blob` now must be valid Blob.\r\n\r\nBlob itself remains sliceable, but slices are also unreadable.\r\n\r\nAll sympthoms remain the same to https://github.com/pouchdb/pouchdb/issues/3932 – I kill DB, create it, replicate from remote, see all files with attaches using Pouch, then close tab, wait a little, reopen it, replicate to and from again (NB: doing nothing in Futon) – and see zero-length attaches using Pouch.\r\n\r\nSomewhere inbetween Pouch index engine works, but don‘t think it changes anything, since, as I understand, views have own DBs.\r\n\r\nUntil I close tab, everything is fine. I cannot even imagine what it can be. Even checked if my SSDs started to degrade – but found no errors.\r\n\r\nThe only specialty is that entire IDB (all objectStores) is quite large – about 1/2Gb. Also one of docs replicated is large JSON (~20Mb) and large attach (also ~20Mb).\r\n\r\nAll steps were performed manually, without any proxy shims or helpers. Sympthoms are same in 3.2.1 and 3.6.0.\r\n\r\nSeems like situation somehow related to https://codereview.chromium.org/865013002\r\n\r\nAs for Opera and FF – very likely that issues I assumed to be same sympthoms, were just aborted replication and too optimistic test, assuming that replication finished succesfully.\r\n\r\nGoing to provide .couch with source DB – to allow repro."}}