Download HTTPS pages with HttpClient 4.5 via proxy in Java
<p>I try to download https pages with Apache HttpClient 4.5 via proxies. </p>

<p>Some download finishes successfully, but with others I get errors. </p>

<p>Why? Do I something wrong?</p>

<pre><code>package hu.istvanbohm.mypublist.test;

import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.concurrent.TimeUnit;

import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.HttpStatus;
import org.apache.http.StatusLine;
import org.apache.http.client.config.CookieSpecs;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.client.StandardHttpRequestRetryHandler;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.apache.http.util.EntityUtils;

public class TestHttpClientDownloader {

    public static void main(String[] args) throws URISyntaxException {
        String rawUrl = "https://*********.com";
        String[] proxies = {"1.161.158.86:8080","1.179.183.27:3128","1.179.201.18:3128","1.207.245.184:80",
                "1.207.62.194:3128","1.234.28.160:80","1.234.28.18:80",
                "1.234.28.20:80","1.234.28.36:80","1.234.28.42:80"};

        URI targetUri = new URI(rawUrl);

        PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
        cm.setMaxTotal(100);

        CloseableHttpClient httpClient = HttpClients.custom()
                .setConnectionManager(cm)
                .setRetryHandler(new StandardHttpRequestRetryHandler(0,false))
                .setConnectionTimeToLive(25, TimeUnit.SECONDS )
                .build();

        for(String proxyString : proxies) {
            String[] proxyParts = proxyString.split(":");
            String proxyUrl = proxyParts[0];
            int proxyPort = Integer.parseInt(proxyParts[1]);

            HttpGet request = new HttpGet(targetUri);
            RequestConfig config = RequestConfig.custom()
                    .setProxy(new HttpHost(proxyUrl,proxyPort))
                    .setCookieSpec(CookieSpecs.STANDARD)
                    .setSocketTimeout(25000)
                    .setConnectTimeout(25000)
                    .setConnectionRequestTimeout(25000)
                    .build();
            request.setConfig(config);
            HttpClientContext context = HttpClientContext.create();
            context.setAttribute("http.protocol.handle-redirects", false);  
            HttpEntity entity = null;
            try {
                System.out.println("Submit: Executing request " + request.getRequestLine() + 
                        " to " + request.getURI()  + " via " + request.getConfig().getProxy());
                CloseableHttpResponse response = httpClient.execute(request,context);
                try {
                    entity = response.getEntity(); 
                    StatusLine statusLine = response.getStatusLine();
                    if(entity!=null &amp;&amp; statusLine.getStatusCode()==HttpStatus.SC_OK) {
                        String result=EntityUtils.toString(entity,org.apache.http.entity.ContentType.getOrDefault(entity).getCharset());
                        if(result!=null &amp;&amp; result.length()&gt;0) System.out.println("Download OK!");
                    } else if(entity!=null){
                        System.out.println("Error - Http status is not OK: " + statusLine.getReasonPhrase());
                    }
                } catch(Exception ex) {
                    System.out.println("Error - HttpClient response: " + ex.getMessage());
                } finally {
                    if(entity!=null) {
                        EntityUtils.consume(entity);
                    }
                    if(response!=null) {
                        response.close();
                    }
                }
            } catch(Exception ex) {
                System.out.println("Error - HttpClient execute: " + ex.getMessage());
            }    
        }

        try {
            httpClient.close();
        } catch (IOException ex) {
            System.out.println("Error - HttpClient close: " + ex.getMessage());
        }
        cm.close();
    }
}
</code></pre>

<p>This is the output that the program generates:</p>

<pre><code>Submit: Executing request GET https://**********.com HTTP/1.1 to https://**********.com via http://1.161.158.86:8080
Error - HttpClient execute: Remote host closed connection during handshake
Submit: Executing request GET https://**********.com HTTP/1.1 to https://**********.com via http://1.179.183.27:3128
Download OK!
Submit: Executing request GET https://**********.com HTTP/1.1 to https://**********.com via http://1.179.201.18:3128
Download OK!
Submit: Executing request GET https://**********.com HTTP/1.1 to https://**********.com via http://1.207.245.184:80
Error - HttpClient execute: Connection reset
Submit: Executing request GET https://**********.com HTTP/1.1 to https://**********.com via http://1.207.62.194:3128
Error - HttpClient execute: Connection reset
Submit: Executing request GET https://**********.com HTTP/1.1 to https://**********.com via http://1.234.28.160:80
Error - HttpClient execute: Unrecognized SSL message, plaintext connection?
Submit: Executing request GET https://**********.com HTTP/1.1 to https://**********.com via http://1.234.28.18:80
Error - HttpClient execute: Unrecognized SSL message, plaintext connection?
Submit: Executing request GET https://**********.com HTTP/1.1 to https://**********.com via http://1.234.28.20:80
Error - HttpClient execute: Unrecognized SSL message, plaintext connection?
Submit: Executing request GET https://**********.com HTTP/1.1 to https://**********.com via http://1.234.28.36:80
Error - HttpClient execute: Unrecognized SSL message, plaintext connection?
Submit: Executing request GET https://**********.com HTTP/1.1 to https://**********.com via http://1.234.28.42:80
Error - HttpClient execute: Unrecognized SSL message, plaintext connection?
</code></pre>

<p>Sample outputs with extended error info and sample url:</p>

<pre><code>Submit: Executing request GET https://en.blog.wordpress.com/ HTTP/1.1 to https://en.blog.wordpress.com/ via http://1.234.28.42:80
2016/08/07 17:42:50:125 CEST [DEBUG] DefaultHttpClientConnectionOperator - Connecting to /1.234.28.42:80
2016/08/07 17:42:50:537 CEST [DEBUG] DefaultHttpClientConnectionOperator - Connection established 192.168.0.9:53098&lt;-&gt;1.234.28.42:80
2016/08/07 17:42:50:950 CEST [DEBUG] MainClientExec - Tunnel to target created.
2016/08/07 17:42:50:955 CEST [DEBUG] SSLConnectionSocketFactory - Enabled protocols: [TLSv1, TLSv1.1, TLSv1.2]
2016/08/07 17:42:50:955 CEST [DEBUG] SSLConnectionSocketFactory - Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2016/08/07 17:42:50:955 CEST [DEBUG] SSLConnectionSocketFactory - Starting handshake
2016/08/07 17:42:51:371 CEST [DEBUG] DefaultManagedHttpClientConnection - http-outgoing-9: Shutdown connection
2016/08/07 17:42:51:371 CEST [DEBUG] MainClientExec - Connection discarded
2016/08/07 17:42:51:371 CEST [DEBUG] PoolingHttpClientConnectionManager - Connection released: [id: 9][route: {tls}-&gt;http://1.234.28.42:80-&gt;https://en.blog.wordpress.com:443][total kept alive: 2; route allocated: 0 of 2; total allocated: 2 of 100]
Error - HttpClient execute: Unrecognized SSL message, plaintext connection?

Submit: Executing request GET https://en.blog.wordpress.com/ HTTP/1.1 to https://en.blog.wordpress.com/ via http://1.207.245.184:80
2016/08/07 17:42:19:867 CEST [DEBUG] RequestAuthCache - Auth cache not set in the context
2016/08/07 17:42:19:867 CEST [DEBUG] PoolingHttpClientConnectionManager - Connection request: [route: {tls}-&gt;http://1.207.245.184:80-&gt;https://en.blog.wordpress.com:443][total kept alive: 2; route allocated: 0 of 2; total allocated: 2 of 100]
2016/08/07 17:42:19:868 CEST [DEBUG] PoolingHttpClientConnectionManager - Connection leased: [id: 3][route: {tls}-&gt;http://1.207.245.184:80-&gt;https://en.blog.wordpress.com:443][total kept alive: 2; route allocated: 1 of 2; total allocated: 3 of 100]
2016/08/07 17:42:19:868 CEST [DEBUG] MainClientExec - Opening connection {tls}-&gt;http://1.207.245.184:80-&gt;https://en.blog.wordpress.com:443
2016/08/07 17:42:19:869 CEST [DEBUG] DefaultHttpClientConnectionOperator - Connecting to /1.207.245.184:80
2016/08/07 17:42:20:353 CEST [DEBUG] DefaultHttpClientConnectionOperator - Connection established 192.168.0.9:53088&lt;-&gt;1.207.245.184:80
2016/08/07 17:42:21:487 CEST [DEBUG] MainClientExec - Tunnel to target created.
2016/08/07 17:42:21:491 CEST [DEBUG] SSLConnectionSocketFactory - Enabled protocols: [TLSv1, TLSv1.1, TLSv1.2]
2016/08/07 17:42:21:492 CEST [DEBUG] SSLConnectionSocketFactory - Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2016/08/07 17:42:21:492 CEST [DEBUG] SSLConnectionSocketFactory - Starting handshake
2016/08/07 17:42:21:509 CEST [DEBUG] DefaultManagedHttpClientConnection - http-outgoing-3: Shutdown connection
2016/08/07 17:42:21:509 CEST [DEBUG] MainClientExec - Connection discarded
2016/08/07 17:42:21:509 CEST [DEBUG] PoolingHttpClientConnectionManager - Connection released: [id: 3][route: {tls}-&gt;http://1.207.245.184:80-&gt;https://en.blog.wordpress.com:443][total kept alive: 2; route allocated: 0 of 2; total allocated: 2 of 100]
Error - HttpClient execute: Connection reset
</code></pre>