{"action":"created","issue":{"url":"https://api.github.com/repos/medic/medic-analytics/issues/7","labels_url":"https://api.github.com/repos/medic/medic-analytics/issues/7/labels{/name}","comments_url":"https://api.github.com/repos/medic/medic-analytics/issues/7/comments","events_url":"https://api.github.com/repos/medic/medic-analytics/issues/7/events","html_url":"https://github.com/medic/medic-analytics/issues/7","id":109831851,"number":7,"title":"full database delete/import is not viable long term","user":{"login":"btbonval","id":880439,"avatar_url":"https://avatars.githubusercontent.com/u/880439?v=3","gravatar_id":"","url":"https://api.github.com/users/btbonval","html_url":"https://github.com/btbonval","followers_url":"https://api.github.com/users/btbonval/followers","following_url":"https://api.github.com/users/btbonval/following{/other_user}","gists_url":"https://api.github.com/users/btbonval/gists{/gist_id}","starred_url":"https://api.github.com/users/btbonval/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/btbonval/subscriptions","organizations_url":"https://api.github.com/users/btbonval/orgs","repos_url":"https://api.github.com/users/btbonval/repos","events_url":"https://api.github.com/users/btbonval/events{/privacy}","received_events_url":"https://api.github.com/users/btbonval/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-10-05T16:06:27Z","updated_at":"2015-11-19T04:08:44Z","closed_at":null,"body":"Presently, the postgres contents are deleted and then the full couchdb is pulled in all over again.\r\n\r\nThis will not be a viable solution for larger projects.\r\n\r\nA better solution would be to grab the list of UUIDs/revs from Couch, see which ones are missing from postgres, and then do a bulk fetch for only those UUIDs/revs that are not already in postgres.\r\n\r\nAdditionally, I have not thus far seen revision used. I need to look more closely to see how to handle that situation. Potentially it might be worth only including the newest revision in postgres, rather than fetching the full history."},"comment":{"url":"https://api.github.com/repos/medic/medic-analytics/issues/comments/157945652","html_url":"https://github.com/medic/medic-analytics/issues/7#issuecomment-157945652","issue_url":"https://api.github.com/repos/medic/medic-analytics/issues/7","id":157945652,"user":{"login":"btbonval","id":880439,"avatar_url":"https://avatars.githubusercontent.com/u/880439?v=3","gravatar_id":"","url":"https://api.github.com/users/btbonval","html_url":"https://github.com/btbonval","followers_url":"https://api.github.com/users/btbonval/followers","following_url":"https://api.github.com/users/btbonval/following{/other_user}","gists_url":"https://api.github.com/users/btbonval/gists{/gist_id}","starred_url":"https://api.github.com/users/btbonval/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/btbonval/subscriptions","organizations_url":"https://api.github.com/users/btbonval/orgs","repos_url":"https://api.github.com/users/btbonval/repos","events_url":"https://api.github.com/users/btbonval/events{/privacy}","received_events_url":"https://api.github.com/users/btbonval/received_events","type":"User","site_admin":false},"created_at":"2015-11-19T04:08:44Z","updated_at":"2015-11-19T04:08:44Z","body":"Step 1: integration testing of partial updates. Need to create a situation where some data is loaded, and then more data is loaded afterwards. Only check that all data is accounted for. Unit testing can check individual cases to ensure existing docs aren't reimported.\r\n\r\nFor integration testing, bulk import into CouchDB might work to build a dataset on top of the existing medic-data, but it'd need to be valid data. https://wiki.apache.org/couchdb/HTTP_Bulk_Document_API#Modify_Multiple_Documents_With_a_Single_Request\r\n\r\nBulk document deletion is addressed above. Perhaps the right way to do integration testing is to extract medic-data, then bulk delete half of it, run an import, then extract medic-data again and run another import and test the data. Problem is that this would succeed with the current integration testing anyway, but instead of update it of postgres would be a full wipe and dump.\r\n"}}