{"action":"closed","issue":{"url":"https://api.github.com/repos/codeforamerica/ohana-api/issues/361","labels_url":"https://api.github.com/repos/codeforamerica/ohana-api/issues/361/labels{/name}","comments_url":"https://api.github.com/repos/codeforamerica/ohana-api/issues/361/comments","events_url":"https://api.github.com/repos/codeforamerica/ohana-api/issues/361/events","html_url":"https://github.com/codeforamerica/ohana-api/issues/361","id":103557710,"number":361,"title":"Difficulty importing large datasets","user":{"login":"cderenburger","id":13293901,"avatar_url":"https://avatars.githubusercontent.com/u/13293901?v=3","gravatar_id":"","url":"https://api.github.com/users/cderenburger","html_url":"https://github.com/cderenburger","followers_url":"https://api.github.com/users/cderenburger/followers","following_url":"https://api.github.com/users/cderenburger/following{/other_user}","gists_url":"https://api.github.com/users/cderenburger/gists{/gist_id}","starred_url":"https://api.github.com/users/cderenburger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cderenburger/subscriptions","organizations_url":"https://api.github.com/users/cderenburger/orgs","repos_url":"https://api.github.com/users/cderenburger/repos","events_url":"https://api.github.com/users/cderenburger/events{/privacy}","received_events_url":"https://api.github.com/users/cderenburger/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":29,"created_at":"2015-08-27T17:31:54Z","updated_at":"2015-09-03T03:03:02Z","closed_at":"2015-09-03T03:03:02Z","body":"I'm attempting to import a whole statewide database and I've run into a few issues.\r\n\r\n1. Importing script fails without an error message if it aborted due to being out of memory or it otherwise did not complete.\r\n2. Importing seems to get exponentially longer the larger the database import.\r\n * Smaller db - 640 organizations, 1250 locations, 2440 services / ~30 minutes to import\r\n    * adding taxonomy and service areas / ~60 minutes to import\r\n * Large db - 5110 organizations, 9920 locations, 20,000 services / 36 hours and counting...\r\n\r\nMy VM is still seems to be working on importing, ruby is using between 2.5-4G memory, 99% cpu.  I had thought it might finish at about 24 hours, but now I'm not sure it will even complete.  Watching <code>top</code> shows the postgres process running only a moment every 5-10 minutes.   \r\n\r\nI'm not sure what my next step needs to be.  I'm wondering about either needing to set up a vagrant 64-bit box or attempting to build and merge dumps like described at http://blog.fespinozacast.com/2014/01/04/merge-two-backups-of-a-postgresql-database/ .\r\n\r\nAny thoughts or suggestions?"}}