{"action":"created","issue":{"url":"https://api.github.com/repos/BookingSync/synced/issues/52","labels_url":"https://api.github.com/repos/BookingSync/synced/issues/52/labels{/name}","comments_url":"https://api.github.com/repos/BookingSync/synced/issues/52/comments","events_url":"https://api.github.com/repos/BookingSync/synced/issues/52/events","html_url":"https://github.com/BookingSync/synced/issues/52","id":120093188,"number":52,"title":"Fetch and save pages separately to avoid keeping all data in memory","user":{"login":"Mandaryn","id":71698,"avatar_url":"https://avatars.githubusercontent.com/u/71698?v=3","gravatar_id":"","url":"https://api.github.com/users/Mandaryn","html_url":"https://github.com/Mandaryn","followers_url":"https://api.github.com/users/Mandaryn/followers","following_url":"https://api.github.com/users/Mandaryn/following{/other_user}","gists_url":"https://api.github.com/users/Mandaryn/gists{/gist_id}","starred_url":"https://api.github.com/users/Mandaryn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Mandaryn/subscriptions","organizations_url":"https://api.github.com/users/Mandaryn/orgs","repos_url":"https://api.github.com/users/Mandaryn/repos","events_url":"https://api.github.com/users/Mandaryn/events{/privacy}","received_events_url":"https://api.github.com/users/Mandaryn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-12-03T04:42:49Z","updated_at":"2015-12-03T09:10:05Z","closed_at":null,"body":"I reach a point when I fetch collections of thousands entries from one endpoint for account.\r\nWith current code https://github.com/BookingSync/synced/blob/master/lib/synced/strategies/full.rb#L172, this is all now firstly fetched to memory and then saved to database, which is bound to cause troubles.\r\n\r\nWe need to be able to save each page separately to database, to avoid bloating memory usage sky high (and running out of memory for larger endpoints)\r\n\r\n@Azdaroth @wijet @Fredar @p55 @ZenCocoon wdyt?"},"comment":{"url":"https://api.github.com/repos/BookingSync/synced/issues/comments/161558653","html_url":"https://github.com/BookingSync/synced/issues/52#issuecomment-161558653","issue_url":"https://api.github.com/repos/BookingSync/synced/issues/52","id":161558653,"user":{"login":"Mandaryn","id":71698,"avatar_url":"https://avatars.githubusercontent.com/u/71698?v=3","gravatar_id":"","url":"https://api.github.com/users/Mandaryn","html_url":"https://github.com/Mandaryn","followers_url":"https://api.github.com/users/Mandaryn/followers","following_url":"https://api.github.com/users/Mandaryn/following{/other_user}","gists_url":"https://api.github.com/users/Mandaryn/gists{/gist_id}","starred_url":"https://api.github.com/users/Mandaryn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Mandaryn/subscriptions","organizations_url":"https://api.github.com/users/Mandaryn/orgs","repos_url":"https://api.github.com/users/Mandaryn/repos","events_url":"https://api.github.com/users/Mandaryn/events{/privacy}","received_events_url":"https://api.github.com/users/Mandaryn/received_events","type":"User","site_admin":false},"created_at":"2015-12-03T09:10:05Z","updated_at":"2015-12-03T09:10:05Z","body":"wrapping all in transaction should not be a problem, rails transactions don't lock tables as far as i know. would probably only fail if 2 syncs for the same are done in parallel, but that would fail anyway now"}}