URLError: <urlopen error [Errno 24] Too many open files>
<p>I'm using scrapy on a dedicated server and I have this error lot of times :</p>

<pre><code>Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
  File "dirbot/spiders/expired.py", line 54, in errback_httpbin
  File "dirbot/spiders/expired.py", line 65, in checkDomain
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
  File "/usr/lib/python2.7/urllib2.py", line 431, in open
  File "/usr/lib/python2.7/urllib2.py", line 449, in _open
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
  File "/usr/lib/python2.7/urllib2.py", line 1227, in http_open
  File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open
URLError: &lt;urlopen error [Errno 24] Too many open files&gt;
</code></pre>

<p>I already add this configuration but I still have this error.</p>

<blockquote>
  <p>at /etc/sysctl.conf</p>
  
  <p>add:</p>

<pre><code>net.core.somaxconn=131072
fs.file-max=131072
</code></pre>
  
  <p>and then: </p>

<pre><code>sudo sysctl -p
</code></pre>
  
  <p>at /usr/include/linux/limits.h </p>
  
  <p>change:</p>

<pre><code>NR_OPEN = 65536
</code></pre>
  
  <p>at /etc/security/limits.conf</p>
  
  <p>add:</p>

<pre><code>*                soft    nofile          65535
*                hard    nofile          65535
</code></pre>
</blockquote>

<p>Can someone help me to fix that ?</p>

<p><strong>Update,
My configuration Scrapy :</strong></p>

<pre><code>custom_settings = {
    'RETRY_ENABLED': False,
    'DEPTH_LIMIT' : 0,
    'DEPTH_PRIORITY' : 1,
    'LOG_ENABLED' : False,
    'CONCURRENT_REQUESTS_PER_DOMAIN' : 64,
    'CONCURRENT_REQUESTS' : 128,
    'REACTOR_THREADPOOL_MAXSIZE' : 30,
    'COOKIES_ENABLED' : False,
    'DOWNLOAD_TIMEOUT' : 10,
    'DOWNLOAD_WARNSIZE' : 66554432
}
</code></pre>