Nginx discards invalid headers when ignore_invalid_headers is set to off
<p>Here's my nginx conf:</p>

<pre><code>http {
    include                        mime.types;
    default_type                   application/octet-stream;
    keepalive_timeout              65;
    server_names_hash_max_size     10000;
    server_names_hash_bucket_size  128;
    client_max_body_size           2m;

    server {
        listen    80;
        server_name     testheader.com;
        ignore_invalid_headers off;
        proxy_http_version 1.1;

        location ~* ^/testheader {
            proxy_request_buffering off;
            proxy_next_upstream off;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Connection "";
            proxy_read_timeout 180s;

            set $upstream 127.0.0.1:8888;
            proxy_pass http://$upstream ;
        }
    }
}
</code></pre>

<p>I sent a sample request as following</p>

<pre><code>POST /testhead/magic?null HTTP/1.1: 
content-type: text/plain; charset=UTF-8
test_trace_id: 1234567890123456
test_span_id: 1234567890123456
x-forwarded-for: 127.0.0.1:8080
x-test-client-host: 127.0.0.1:8888
x-test-request.toplevel.uuid: xxxxxxx-xxxxxx-xxxxxx-xxxxxxx
accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
accept-encoding: gzip
connection: keep-alive
accept-language: null
host: top.ctrip.uat.qa.nt.ctripcorp.com
user-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0
Content-Length: 264


{"ActCode":2}
</code></pre>

<p>I expected to get all the invalid headers like <strong>test_trace_id, test_span_id, all x-{$value}</strong> from my upstream server 127.0.0.1:8888, but <strong>none of the headers was shown</strong>. In the mean time, nginx log didn't report any errors in sending invalid headers phase.</p>

<p>I also tried capturing requests at the target upstream server using tcpdump, and I'm for sure nginx did not even try to send them out.</p>

<p>However, in another sample:</p>

<pre><code>POST /testhead/magic?null HTTP/1.1: 
content-type: text/plain; charset=UTF-8
test_trace_id: 1234567890123456
test_span_id: 1234567890123456
host: top.ctrip.uat.qa.nt.ctripcorp.com
x-forwarded-for: 127.0.0.1:8080
x-test-request.toplevel.uuid: xxxxxxx-xxxxxx-xxxxxx-xxxxxxx
accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
accept-encoding: gzip
connection: keep-alive
accept-language: null
user-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0
Content-Length: 264


{"ActCode":3}
</code></pre>

<p>The invalid header <strong>x-test-request.toplevel.uuid</strong> was sent to the upstream server.</p>

<p>I don't think the missing two headers in the second sample is the cause of the issue. I tried many requests and it seems like this problem happens randomly, sometimes the directive works perfectly, sometimes it doesn't.</p>

<p>Is it a bug?
The config file clearly tells nginx to ignore invalid headers and pass whatever it gets from client to the upstream. Nginx has no reason to throw those headers away.</p>

<p>Does anyone know how to fix it?
Thanks in advance.</p>