How do you optimise memory/cpu on NodeJS / Express?
<p>I have a Node app, built on Express, that uses a web scraper to load and parse data.</p>

<p>I've read a lot about NodeJS's scaleability and being able to handle a heap of concurrent connections, however when you're running a web scraper (sending off 1000+ concurrent requests) I feel like things start to crumble a bit.</p>

<p>When running, my server is unresponsive to other API requests, and running several instances at once causes things to slow down to a snail's pace.</p>

<p>I can't find any documentation on what the limits are, what they should be, how many requests I should be pooling together and so on. </p>

<p>Should I be limiting my scraper's requests to 10 per second? 100 per second? 1000 per second? Or could I perhaps be increasing the amount of CPU/memory allocated to my node process on my VPS?</p>

<p>EDIT: For those voting to close because this question is too opinion based, here's concretely what I'm asking:</p>

<ol>
<li>How many HTTP requests can an Express app simultaneously perform before it starts to hit performance</li>
<li>Does increasing memory / cpu available to the app help in any way?</li>
</ol>