Best performance for Scrapy
<p>I'm using Scrapy on my dedicated server, I would like to know how get the best performance for my crawler.</p>

<p>Here is my custom settings : </p>

<pre><code>custom_settings = {
    'RETRY_ENABLED': True,
    'DEPTH_LIMIT' : 0,
    'DEPTH_PRIORITY' : 1,
    'LOG_ENABLED' : False,
    'CONCURRENT_REQUESTS_PER_DOMAIN' : 32,
    'CONCURRENT_REQUESTS' : 64,
}
</code></pre>

<p>I actually crawl around 200 link/minutes.</p>

<p>Server :</p>

<pre><code>32 Go RAM : DDR4 ECC 2133 MHz
CPU : 4c/8t : 2,2 / 2,6 GHz
</code></pre>