A method for very efficient (fast) network polling in C
<p><strong>Short version</strong>: I don’t want to use third party libraries or framesworks like Netmap or DPDK, is there anything faster than <code>poll()</code> or <code>select()</code>, or can I make those calls more efficient?</p>

<p><strong>Full version</strong>: I have a single threaded application which uses a single socket to send data as fast as possible and I'm using valgrind/cachegrind/callgrind to try and increase the efficiency (and thus, throughput of the application).</p>

<p>At present the receiving host sites in an infinite loop trying to check received data as fast as possible (which needs to be non-blocking, when there is no packet to process "other stuff" is done). I am using <code>select()</code> on the receiving host because <code>select()</code> uses a timeval which offers microsecond polling frequency whereas <code>poll()</code> uses a millisecond value. Callgrind is showing me that <code>FD_SET()</code> has made nearly twice as many instruction executions as the <code>select()</code> operation, and one downside to <code>select()</code> is that I must run <code>FD_SET()</code> every  iteration of the receive loop:</p>

<pre><code>         .    .    .          .     .    .          .     .    .              // Poll for incoming frames
 4,361,236    1    1  2,180,618     0    0  2,180,618     0    0              TEST_INTERFACE-&gt;TV_SELECT_DELAY.tv_sec = 0;
 4,361,236    0    0  2,180,618     0    0  2,180,618     0    0              TEST_INTERFACE-&gt;TV_SELECT_DELAY.tv_usec = 000000;
56,696,068    2    2 15,264,326     0    0  2,180,618     0    0              FD_SET(TEST_INTERFACE-&gt;SOCKET_FD, &amp;TEST_INTERFACE-&gt;FD_READS);
         .    .    .          .     .    .          .     .    .  
         .    .    .          .     .    .          .     .    .              TEST_INTERFACE-&gt;SELECT_RET_VAL = select(TEST_INTERFACE-&gt;SOCKET_FD_COUNT,
         .    .    .          .     .    .          .     .    .                                                      &amp;TEST_INTERFACE-&gt;FD_READS,
         .    .    .          .     .    .          .     .    .                                                      NULL, NULL,
30,528,652    1    1 10,903,090     0    0 15,264,326     0    0                                                      &amp;TEST_INTERFACE-&gt;TV_SELECT_DELAY);
         .    .    .          .     .    .          .     .    .  
         .    .    .          .     .    .          .     .    .  
 9,432,052    0    0  4,361,236     0    0          0     0    0              if (TEST_INTERFACE-&gt;SELECT_RET_VAL &gt; 0 &amp;&amp;
 7,450,590    1    1  2,128,740     0    0          0     0    0                  FD_ISSET(TEST_INTERFACE-&gt;SOCKET_FD, &amp;TEST_INTERFACE-&gt;FD_READS))
         .    .    .          .     .    .          .     .    .              {
         .    .    .          .     .    .          .     .    .  
         .    .    .          .     .    .          .     .    .                  RX_LEN = recvfrom(TEST_INTERFACE-&gt;SOCKET_FD,
         .    .    .          .     .    .          .     .    .                                    FRAME_HEADERS-&gt;RX_BUFFER,
         .    .    .          .     .    .          .     .    .                                    TEST_PARAMS-&gt;F_SIZE_TOTAL,
 5,321,850    1    1  2,128,740     0    0  2,838,320     0    0                                    0, NULL, NULL);
         .    .    .          .     .    .          .     .    .  
</code></pre>

<p>I am getting an average of 130-140Mpbs of receive throughput using <code>select()</code>. Using <code>poll()</code> I am getting an average of 150-160Mbps.</p>

<pre><code>         .    .    .         .     .    .         .     .    .              // Poll for incoming frames
7,347,032    2    2 1,836,758     0    0 4,591,895     0    0  TEST_INTERFACE-&gt;SELECT_RET_VAL = poll(TEST_INTERFACE-&gt;fds, 1, 0);
         .    .    .         .     .    .         .     .    .  
         .    .    .         .     .    .         .     .    .  
 3,673,516    0    0 1,836,758     0    0         0     0    0  if (TEST_INTERFACE-&gt;SELECT_RET_VAL &gt; 0)
         .    .    .         .     .    .         .     .    .  {
         .    .    .         .     .    .         .     .    .  
 2,142,186    0    0   714,062     0    0         0     0    0      if ( TEST_INTERFACE-&gt;fds[0].revents &amp; POLLIN )
   714,062    0    0   357,031     0    0   357,031     0    0           TEST_INTERFACE-&gt;fds[0].revents = 0;
         .    .    .         .     .    .         .     .    .  
         .    .    .         .     .    .         .     .    .                  RX_LEN = recvfrom(TEST_INTERFACE-&gt;SOCKET_FD,
         .    .    .         .     .    .         .     .    .                                    FRAME_HEADERS-&gt;RX_BUFFER,
         .    .    .         .     .    .         .     .    .                                    TEST_PARAMS-&gt;F_SIZE_TOTAL,
 5,355,465    1    1 2,142,186     0    0 2,856,248     0    0                                    0, NULL, NULL);
</code></pre>

<p>With <code>poll()</code> I am passing a timeout value of <code>0</code> so hopefully that is not blocking at all for anytime, but I'm not sure if that is correct and it is actually faster than <code>select()</code>.</p>

<p>In the lab I have a couple of servers back to back with 10Gbps NICs. Using <code>select()</code> with the Tx hosting sending data at line rate 10Gbps the Rx host can process about 9.5Gbps of traffic. The above results are just between to virtual machines on my laptop (the lab is out of action at present) to show that <code>poll()</code> is giving me a slight increase on the Rx host. In the above results we can see that <code>poll()</code> was faster however 10Gbps is a packet every few tens of nano-seconds so I’m not sure (when the lab is working again) if I will see any improvement.</p>

<p>So I guess I have two questions: </p>

<ol>
<li><p>In my <code>poll()</code> vs <code>select()</code> tests, since I am using a single socket <code>poll()</code> is slightly faster but will this hold true on faster connections, 10/20/30/40Gbps, or at those speed would <code>select()</code> be faster?</p></li>
<li><p>Since I am only using a single socket, is there any speed increase if I switched away from <code>select()</code>/<code>poll()</code> to <code>epoll()</code>? Or is there some other “built in” method I can use, that is faster than <code>poll()</code> and <code>select()</code>, or something I can do to reduce their execution time?</p></li>
</ol>