PHP - `get_headers` returns "400 Bad Request" and "403 Forbidden" for valid URLs?
<p><strong>Working solution at bottom of description!</strong></p>
<p>I am running PHP 5.4, and trying to get the headers of a list of URLs.</p>
<p>For the most part, everything is working fine, but there are three URLs that are causing issues (and likely more, with more extensive testing).</p>
<pre><code>'http://www.alealimay.com'
'http://www.thelovelist.net'
'http://www.bleedingcool.com'
</code></pre>
<p>All three sites work fine in a browser, and produce the following header responses:</p>
<p><em>(From Safari)</em></p>
<p><a href="https://i.stack.imgur.com/zOC3Q.png" rel="noreferrer"><img src="https://i.stack.imgur.com/zOC3Q.png" alt="successful headers" /></a></p>
<p>Note that all three header responses are <code>Code = 200</code></p>
<p>But retrieving the headers via PHP, using <a href="http://php.net/manual/en/function.get-headers.php" rel="noreferrer"><code>get_headers</code></a>...</p>
<pre><code>stream_context_set_default(array('http' =&gt; array('method' =&gt; &quot;HEAD&quot;)));
$headers = get_headers($url, 1);
stream_context_set_default(array('http' =&gt; array('method' =&gt; &quot;GET&quot;)));
</code></pre>
<p>... returns the following:</p>
<pre><code>url  ......  &quot;http://www.alealimay.com&quot;

headers
|    0  ............................  &quot;HTTP/1.0 400 Bad Request&quot;
|    content-length  ...............  &quot;378&quot;
|    X-Synthetic  ..................  &quot;true&quot;
|    expires  ......................  &quot;Thu, 01 Jan 1970 00:00:00 UTC&quot;
|    pragma  .......................  &quot;no-cache&quot;
|    cache-control  ................  &quot;no-cache, must-revalidate&quot;
|    content-type  .................  &quot;text/html; charset=UTF-8&quot;
|    connection  ...................  &quot;close&quot;
|    date  .........................  &quot;Wed, 24 Aug 2016 01:26:21 UTC&quot;
|    X-ContextId  ..................  &quot;QIFB0I8V/xsTFMREg&quot;
|    X-Via  ........................  &quot;1.0 echo109&quot;
   


url  ......  &quot;http://www.thelovelist.net&quot;

headers
|    0  ............................  &quot;HTTP/1.0 400 Bad Request&quot;
|    content-length  ...............  &quot;378&quot;
|    X-Synthetic  ..................  &quot;true&quot;
|    expires  ......................  &quot;Thu, 01 Jan 1970 00:00:00 UTC&quot;
|    pragma  .......................  &quot;no-cache&quot;
|    cache-control  ................  &quot;no-cache, must-revalidate&quot;
|    content-type  .................  &quot;text/html; charset=UTF-8&quot;
|    connection  ...................  &quot;close&quot;
|    date  .........................  &quot;Wed, 24 Aug 2016 01:26:22 UTC&quot;
|    X-ContextId  ..................  &quot;aNKvf2RB/bIMjWyjW&quot;
|    X-Via  ........................  &quot;1.0 echo103&quot;



url  ......  &quot;http://www.bleedingcool.com&quot;

headers
|    0  ............................  &quot;HTTP/1.1 403 Forbidden&quot;
|    Server  .......................  &quot;Sucuri/Cloudproxy&quot;
|    Date  .........................  &quot;Wed, 24 Aug 2016 01:26:22 GMT&quot;
|    Content-Type  .................  &quot;text/html&quot;
|    Content-Length  ...............  &quot;5311&quot;
|    Connection  ...................  &quot;close&quot;
|    Vary  .........................  &quot;Accept-Encoding&quot;
|    ETag  .........................  &quot;\&quot;57b7f28e-14bf\&quot;&quot;
|    X-XSS-Protection  .............  &quot;1; mode=block&quot;
|    X-Frame-Options  ..............  &quot;SAMEORIGIN&quot;
|    X-Content-Type-Options  .......  &quot;nosniff&quot;
|    X-Sucuri-ID  ..................  &quot;11005&quot;
</code></pre>
<p>This is the case regardless of changing the stream_context</p>
<pre><code>//stream_context_set_default(array('http' =&gt; array('method' =&gt; &quot;HEAD&quot;)));
$headers = get_headers($url, 1);
//stream_context_set_default(array('http' =&gt; array('method' =&gt; &quot;GET&quot;)));
</code></pre>
<p>Produces the same result.</p>
<p>No warnings or errors are thrown for any of these (normally have the errors suppressed with <code>@get_headers</code>, but there is no difference either way).</p>
<p>I have checked my <code>php.ini</code>, and have <a href="http://php.net/manual/en/filesystem.configuration.php#ini.allow-url-fopen" rel="noreferrer"><code>allow_url_fopen</code></a> set to <code>On</code>.</p>
<p>I am headed towards <a href="https://stackoverflow.com/questions/39111770/php-differences-between-get-headers-and-stream-get-meta-data"><code>stream_get_meta_data</code></a>, <em>and am not interested in <code>CURL</code> solutions</em>. <code>stream_get_meta_data</code> (and its accompanying <code>fopen</code>) will fail in the same spot as <code>get_headers</code>, so fixing one will fix both in this case.</p>
<p>Usually, if there are redirects, the output looks like:</p>
<pre><code>url  ......  &quot;http://www.startingURL.com/&quot;

headers
|    0  ............................  &quot;HTTP/1.1 301 Moved Permanently&quot;
|    1  ............................  &quot;HTTP/1.1 200 OK&quot;
|    Date
|    |    &quot;Wed, 24 Aug 2016 02:02:29 GMT&quot;
|    |    &quot;Wed, 24 Aug 2016 02:02:32 GMT&quot;
|    
|    Server
|    |    &quot;Apache&quot;
|    |    &quot;Apache&quot;
|    
|    Location  .....................  &quot;http://finishingURL.com/&quot;
|    Connection
|    |    &quot;close&quot;
|    |    &quot;close&quot;
|    
|    Content-Type
|    |    &quot;text/html; charset=UTF-8&quot;
|    |    &quot;text/html; charset=UTF-8&quot;
|    
|    Link  .........................  &quot;; rel=\&quot;https://api.w.org/\&quot;, ; rel=shortlink&quot;
</code></pre>
<p><strong>How come the sites work in browsers, but fail when using <code>get_headers</code>?</strong></p>
<p>There are various SO posts discussing the same thing, but the solution for all of them doesn't pertain to this case:</p>
<p><a href="https://stackoverflow.com/questions/36916404/http-request-failed-in-php-bad-request?answertab=votes#tab-top"><code>POST</code> requires <code>Content-Length</code></a> (I'm sending a <code>HEAD</code> request, no content is returned)</p>
<p><a href="https://stackoverflow.com/questions/30576588/php-get-headers-return-wrong-result-when-non-utf-symbols-are-in-url?answertab=votes#tab-top">URL contains UTF-8 data</a> (The only chars in these URLs are all from the Latin alphabet)</p>
<p><a href="https://stackoverflow.com/questions/697472/why-file-get-contents-returns-failed-to-open-stream-http-request-failed#answer-697498">Cannot send a URL with spaces in it</a> (These URLs are all space-free, and very ordinary in every way)</p>
<h1>Solution!</h1>
<p><em>(Thanks to Max in the answers below for pointing me on the right track.)</em></p>
<p>The issue is because there is no pre-defined <code>user_agent</code>, without either setting on in <code>php.ini</code>, or declaring it in code.</p>
<p>So, I change the <code>user_agent</code> to mimic a browser, do the deed, and then revert it back to stating value (likely blank).</p>
<pre><code>$OriginalUserAgent = ini_get('user_agent');
ini_set('user_agent', 'Mozilla/5.0');

$headers = @get_headers($url, 1);

ini_set('user_agent', $OriginalUserAgent);
</code></pre>
<p>User agent change found <a href="https://stackoverflow.com/questions/8403025/how-to-set-user-agent-for-get-headers-php-function">here</a>.</p>