Python performance - best parallelism approach
<p>I am implementing a Python script that needs to keep sending 1500+ packets in parallel in less than 5 seconds each.</p>

<p>In a nutshell what I need is:</p>

<pre><code>def send_pkts(ip):
    #craft packet
    while True:
        #send packet
        time.sleep(randint(0,3))

for x in list[:1500]:
    send_pkts(x)
    time.sleep(randint(1,5))
</code></pre>

<p>I have tried the simple single-threaded, multithreading, multiprocessing and multiprocessing+multithreading forms and had the following issues:</p>

<ol>
<li>Simple single-threaded:
The "<em>for</em> delay" seems to compromise the "5 seconds" dependency.</li>
<li>Multithreading:
I think I could not accomplish what I desire due to Python GIL limitations.</li>
<li>Multiprocessing:
That was the best approach that seemed to work. However, due to excessive quantity of process the VM where I am running the script freezes (of course, 1500 process running). Thus becoming impractical.</li>
<li>Multiprocessing+Multithreading:
In this approach I created less process with each of them calling some threads (lets suppose: 10 process calling 150 threads each). It was clear that the VM is not freezing as fast as approach number 3, however the most "concurrent packet sending" I could reach was ~800. GIL limitations? VM limitations?
In this attempt I also tried using Process Pool but the results where similar.</li>
</ol>

<p>Is there a better approach I could use to accomplish this task?</p>

<p>[1] EDIT 1:</p>

<pre><code> def send_pkt(x):
     #craft pkt
     while True:
         #send pkt
         gevent.sleep(0)

 gevent.joinall([gevent.spawn(send_pkt, x) for x in list[:1500]])
</code></pre>

<p>[2] EDIT 2 (gevent monkey-patching):</p>

<pre><code>from gevent import monkey; monkey.patch_all()

jobs = [gevent.spawn(send_pkt, x) for x in list[:1500]]
gevent.wait(jobs)
#for send_pkt(x) check [1]
</code></pre>

<p>However I got the following error: <strong>"ValueError: filedescriptor out of range in select()"</strong>. So I checked my system ulimit (Soft and Hard both are maximum: 65536).
After, I checked it has something to do with <em>select()</em> limitations over Linux (1024 fds maximum). Please check: <a href="http://man7.org/linux/man-pages/man2/select.2.html" rel="noreferrer">http://man7.org/linux/man-pages/man2/select.2.html</a> (BUGS section) - In orderto overcome that I should use <em>poll()</em> (<a href="http://man7.org/linux/man-pages/man2/poll.2.html" rel="noreferrer">http://man7.org/linux/man-pages/man2/poll.2.html</a>) instead. But with <em>poll()</em> I return to same limitations: as polling is a "blocking approach".</p>

<p>Regards,</p>