High latency after transferring Lua codebase from custom HTTP server to Nginx
<p>For the past few years we have been running a custom, pre-forking, HTTP server written in C++, libevent and embedded Lua (LuaJit using the exact same LuaJit shared library that openresty, an Nginx package, uses). It's got almost the same architecture as Nginx just by coincidence. </p>

<p>Most of the business functionality of this server is in the Lua script "handlers" and when we saw Nginx supported Lua we thought we'd move our code base over to Nginx and take advantage of all the community support that Nginx has. We got all the code moved over and our testing shows everything working, as far as the functionality being preserved, however....</p>

<p>We have run into a problem. The latency for an HTTP request on the old custom, C++ server was typically 2ms round trip. The new server based on Nginx is up around 60ms. <em>The weird thing is, our internal measurements of the actual Lua code executing (using Lua os.clock() to time code regions) shows the two server versions are virtually identical as far as Lua execution speed.</em> </p>

<p>Using various profilers doesn't show any extremely large area of slowness on the Nginx code. pcre is the highest with about 6% of the time spent.</p>

<p>One more thing. The setup we have has a Proxy Nginx server talking to our Nginx Lua Server, both on the same machine, talking over 127.0.0.1, so it's LAN => Nginx Proxy => Nginx Lua Server. </p>

<pre><code>ldd nginx
    linux-vdso.so.1 =&gt;  (0x00007ffc02ff3000)
    libpthread.so.0 =&gt; /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f88d0b46000)
    libcrypt.so.1 =&gt; /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f88d090f000)
    libluajit-5.1.so.2 =&gt; /d1/apps/mt3/openresty/luajit/lib/libluajit-5.1.so.2 (0x00007f88d069a000)
    libm.so.6 =&gt; /lib/x86_64-linux-gnu/libm.so.6 (0x00007f88d0418000)
    libpcre.so.3 =&gt; /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f88d01db000)
    libssl.so.1.0.0 =&gt; /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f88cff79000)
    libcrypto.so.1.0.0 =&gt; /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f88cfb81000)
    libdl.so.2 =&gt; /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f88cf97d000)
    libz.so.1 =&gt; /lib/x86_64-linux-gnu/libz.so.1 (0x00007f88cf765000)
    libc.so.6 =&gt; /lib/x86_64-linux-gnu/libc.so.6 (0x00007f88cf3da000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f88d0d72000)
    libgcc_s.so.1 =&gt; /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f88cf1c4000)
</code></pre>

<p>Anyone got any ideas?</p>