{"action":"created","issue":{"url":"https://api.github.com/repos/vitaly-t/pg-promise/issues/50","labels_url":"https://api.github.com/repos/vitaly-t/pg-promise/issues/50/labels{/name}","comments_url":"https://api.github.com/repos/vitaly-t/pg-promise/issues/50/comments","events_url":"https://api.github.com/repos/vitaly-t/pg-promise/issues/50/events","html_url":"https://github.com/vitaly-t/pg-promise/issues/50","id":110541240,"number":50,"title":"INSERT batch w/no transaction","user":{"login":"paulxtiseo","id":2977050,"avatar_url":"https://avatars.githubusercontent.com/u/2977050?v=3","gravatar_id":"","url":"https://api.github.com/users/paulxtiseo","html_url":"https://github.com/paulxtiseo","followers_url":"https://api.github.com/users/paulxtiseo/followers","following_url":"https://api.github.com/users/paulxtiseo/following{/other_user}","gists_url":"https://api.github.com/users/paulxtiseo/gists{/gist_id}","starred_url":"https://api.github.com/users/paulxtiseo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/paulxtiseo/subscriptions","organizations_url":"https://api.github.com/users/paulxtiseo/orgs","repos_url":"https://api.github.com/users/paulxtiseo/repos","events_url":"https://api.github.com/users/paulxtiseo/events{/privacy}","received_events_url":"https://api.github.com/users/paulxtiseo/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/vitaly-t/pg-promise/labels/question","name":"question","color":"cc317c"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-10-08T21:07:31Z","updated_at":"2015-10-13T16:20:34Z","closed_at":"2015-10-13T16:20:34Z","body":"I have a function that processes large CSV data files (>1GB). The code below errors out due to memory exhaustion. Basically, I'm creating a promise (db.none) asynchronously for each CSV record which is not good. Any recommendations? In my case, I don't need transaction support, although I do want to report on failed rows. \r\n\r\n```javascript\r\nreturn promise.map(this.mappedFiles, function(file) {\r\n  var stream = fs.createReadStream(file.file_name);\r\n  return api.parsers.processCsvStream(stream, function(data) {\r\n    return db.none('INSERT ...', data).then(function(result) { return result; });\r\n  })\r\n}\r\n```\r\n\r\n```javascript\r\napi.parsers.processCsvStream = function(passedStream, processor) {\r\n  if(!(passedStream instanceof stream.Stream)) {\r\n    return promise.reject('No passedStream received.');\r\n  }\r\n\r\n  if(!_.isFunction(processor)) {\r\n    return promise.reject('No processor received.');\r\n  }\r\n\r\n  var parser = csv.parse(passedStream, {trim: true});\r\n\r\n  parser.on('readable', function() {\r\n    var data;\r\n    while(data = parser.read()) {\r\n      processor(data);\r\n    }\r\n  })\r\n\r\n  passedStream.pipe(parser);\r\n\r\n  return new Promise(function(resolve, reject) {\r\n    parser.on('end', resolve);\r\n    parser.on('error', reject);\r\n  });\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/vitaly-t/pg-promise/issues/comments/147766577","html_url":"https://github.com/vitaly-t/pg-promise/issues/50#issuecomment-147766577","issue_url":"https://api.github.com/repos/vitaly-t/pg-promise/issues/50","id":147766577,"user":{"login":"vitaly-t","id":5108906,"avatar_url":"https://avatars.githubusercontent.com/u/5108906?v=3","gravatar_id":"","url":"https://api.github.com/users/vitaly-t","html_url":"https://github.com/vitaly-t","followers_url":"https://api.github.com/users/vitaly-t/followers","following_url":"https://api.github.com/users/vitaly-t/following{/other_user}","gists_url":"https://api.github.com/users/vitaly-t/gists{/gist_id}","starred_url":"https://api.github.com/users/vitaly-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vitaly-t/subscriptions","organizations_url":"https://api.github.com/users/vitaly-t/orgs","repos_url":"https://api.github.com/users/vitaly-t/repos","events_url":"https://api.github.com/users/vitaly-t/events{/privacy}","received_events_url":"https://api.github.com/users/vitaly-t/received_events","type":"User","site_admin":false},"created_at":"2015-10-13T16:20:34Z","updated_at":"2015-10-13T16:20:34Z","body":"Your task has just become even easier, because I released version [2.0.11](https://github.com/vitaly-t/pg-promise/releases/tag/v.2.0.11) that adds support for `spex.page` natively.\r\n\r\nAt this point I suggest that you try and come up with a solution based on the methods available to you, which is more than enough. And if you are stuck along the way with something more specific, you can open an issue accordingly, and I will try to help you.\r\n\r\nIn the meantime, this issue here is too broad to make any progress, so I'm closing it.\r\n"}}