GuzzleHTTP Encoding Problems
<p>im trying to use GuzzleHTTP 6 for web scraping, and so far, im not able to find a solution for messedup encoding in response body.</p>
<p>Lets say i want to parse the web page, which returs data in many different languages.</p>
<p><strong>Client Initialization</strong></p>
<pre><code>public function __construct() {
    $this-&gt;dataClient = new Client(['base_uri' =&gt; 'http://somewebsite.org/{language_code}']);
}
</code></pre>
<p><strong>Using Data Client</strong></p>
<pre><code>$request = $this-&gt;dataClient-&gt;get('/endpoint/' . $data_query . '/');
$response = $request-&gt;getBody()-&gt;__toString();
$decoded = json_decode($response, true);
foreach ($decoded as $index =&gt; $data) {
    $decoded[$index] = str_replace(['&lt;option', '&gt;', '&lt;/option&gt;'], '', $data);
}
return $decoded;
</code></pre>
<p><strong>Problems:</strong></p>
<ol>
<li><p>If text is in English, response looks almost fine, except that some of the characters are messed up</p>
<p>manipulation, thereâ;€™s</p>
</li>
</ol>
<p>Instead of</p>
<pre><code>manipulation, there's
</code></pre>
<ol start="2">
<li><p>If i'm trying to get data in any other languages, thats what i get (data in Russian)</p>
<p>Ð;Ð;°; Ð;¿;Ð;µ;Ñ;€Ð;²;Ñ;‹Ð;¹; Ð;²;Ð</p>
</li>
</ol>
<p>Instead of</p>
<pre><code>На первый взгляд
</code></pre>
<p>And the problem is, if you view website, its all fine and nice, but if you try to scrape it, you will face these problems.
So far, im not able to find the source of the problem, neither utf8_decode, or iconv helps me to solve the problem.<br />
Any solutions are highly welcome!</p>
<p><strong>So, here is a small update</strong>
This is the parsing function:</p>
<pre><code>public function processData($data_query) {
    $request = $this-&gt;dataClient-&gt;get('/endpoint/' . $data_query . '/');
    $response = $request-&gt;getBody()-&gt;__toString();
    // echo $response; - Everything is fine, no encoding problems
    // return $response; - Encoding problems
    $decoded = json_decode($response, true);
    // return $decoded; - Encoding problems
    foreach ($decoded as $index =&gt; $data) {
        $decoded[$index] = str_replace(['&lt;option', '&gt;', '&lt;/option&gt;'], '', $data);
    }
    return $decoded; - Encoding Problems
}
</code></pre>
<p><strong>Raw response headers</strong></p>
<pre><code>{
    Date: [
        &quot;Wed, 08 Jun 2016 01:45:30 GMT&quot;
    ],
    Server: [
        &quot;Apache&quot;
    ],
    X-Frame-Options: [
        &quot;SAMEORIGIN&quot;
    ],
    Retry-After: [
        &quot;600&quot;
    ],
    Content-Language: [
        &quot;en-GB&quot;
    ],
    Vary: [
        &quot;Accept-Encoding&quot;
    ],
    Transfer-Encoding: [
        &quot;chunked&quot;
    ],
    Content-Type: [
        &quot;text/html;charset=UTF-8&quot;
    ]
}
</code></pre>