Perl LWP::UserAgent, https proxy to specific https

Question

Read all I could find about subject... can't figure how make it work.

I use freshly updated (09 Jan 2015) Active State Perl, and a standard approach

my $ua=LWP::UserAgent->new();  
$ua->agent('Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/33.0');
$ua->timeout(120);
$ua->cookie_jar( {} );
$ua->proxy(['https'],'https://199.200.120.140:8089') #taken from http://proxylist.hidemyass.com/search-1308872#listable
my $response = $ua->get('https://www.comparis.ch');   #this web site I struggle with, I can connect to https://github.com for example... But I also can connect to comparis.ch through same proxy using urllib3 on Python3.4... 

If somebody can explain how to get 200 from the web site? The output from

The response returns either Timeout (if I use http proxy, or

LWP::Protocol::https::Socket: SSL connect attempt failed error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol at E:/Perl64/lib/LWP/Protocol/http.pm line 49." 

If I try

LWP::UserAgent->new(ssl_opts => { verify_hostname => 0, SSL_version => 'SSLv3' });

Then I getting

LWP::Protocol::https::Socket: SSL connect attempt failed error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number at E:/Perl64/lib/LWP/Protocol/http.pm line 49.

I think that web site timeouting connections from some regions, so have to use proxy from the USA or Western Europe.

As requested I added output for debug (for simple UserAgent->new() -- no options), it tries two proxies 192.3.121.204:3128 and 64.31.22.131:7808 from http://proxylist.hidemyass.com/search-1305502#listable. The 1st seems like just timed out (most likely was blocked) but second shows different behavior.

perl -MIO::Socket::SSL=debug4 test.pl

DEBUG: .../IO/Socket/SSL.pm:2555: new ctx 61267216
DEBUG: .../IO/Socket/SSL.pm:539: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:1769: IO::Socket::INET configuration failed
DEBUG: .../IO/Socket/SSL.pm:2588: free ctx 61267216 open=61267216
DEBUG: .../IO/Socket/SSL.pm:2593: free ctx 61267216 callback
DEBUG: .../IO/Socket/SSL.pm:2600: OK free ctx 61267216
500 Can't connect to 192.3.121.204:3128 (10060)
Content-Type: text/plain
Client-Date: Fri, 09 Jan 2015 10:15:18 GMT
Client-Warning: Internal response

Can't connect to 192.3.121.204:3128 (10060)

LWP::Protocol::https::Socket: connect: 10060 at E:/Perl64/lib/LWP/Protocol/http.pm line 49.

DEBUG: .../IO/Socket/SSL.pm:2555: new ctx 61267216
DEBUG: .../IO/Socket/SSL.pm:539: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:541: socket connected
DEBUG: .../IO/Socket/SSL.pm:563: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:599: not using SNI because hostname is unknown
DEBUG: .../IO/Socket/SSL.pm:650: set socket to non-blocking to enforce timeout=120
DEBUG: .../IO/Socket/SSL.pm:663: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:673: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:683: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:703: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:663: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:1780: SSL connect attempt failed

DEBUG: .../IO/Socket/SSL.pm:1785: SSL connect attempt failed error:140770FC:SSL     routines:SSL23_GET_SERVER_HELLO:unknown protocol
DEBUG: .../IO/Socket/SSL.pm:669: fatal SSL error: SSL connect attempt failed error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
DEBUG: .../IO/Socket/SSL.pm:1769: IO::Socket::INET configuration failed
DEBUG: .../IO/Socket/SSL.pm:2588: free ctx 61267216 open=61267216
DEBUG: .../IO/Socket/SSL.pm:2593: free ctx 61267216 callback
DEBUG: .../IO/Socket/SSL.pm:2600: OK free ctx 61267216
500 Can't connect to 64.31.22.131:7808
Content-Type: text/plain
Client-Date: Fri, 09 Jan 2015 10:15:19 GMT
Client-Warning: Internal response

Can't connect to 64.31.22.131:7808

LWP::Protocol::https::Socket: SSL connect attempt failed error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol at E:/Perl64/lib/LWP/Protocol/http.pm line 49.

After Stefan suggestion I used http:// for proxy address (and not https) and this produced a little bit better results as follows, also, it perfectly works under Ubuntu, but not in windows...

DEBUG: .../IO/Socket/SSL.pm:2555: new ctx 65664688
DEBUG: .../IO/Socket/SSL.pm:1354: start handshake
DEBUG: .../IO/Socket/SSL.pm:563: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:599: not using SNI because hostname is unknown
DEBUG: .../IO/Socket/SSL.pm:650: set socket to non-blocking to enforce timeout=120
DEBUG: .../IO/Socket/SSL.pm:663: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:673: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:683: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:703: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:663: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:673: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:683: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:703: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:663: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:673: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:683: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:703: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:663: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:673: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:683: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:703: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:2411: ok=1 cert=69181312
DEBUG: .../IO/Socket/SSL.pm:2411: ok=1 cert=69943488
DEBUG: .../IO/Socket/SSL.pm:2411: ok=1 cert=69943136
DEBUG: .../IO/Socket/SSL.pm:1559: scheme=www cert=69943136
DEBUG: .../IO/Socket/SSL.pm:1569: identity=www.comparis.ch cn=www.comparis.ch alt=2 www.comparis.ch 2 it.comparis.ch 2 en.comparis.ch 2 fr.comparis.ch
DEBUG: .../IO/Socket/SSL.pm:663: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:673: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:683: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:703: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:663: Net::SSLeay::connect -> 1
DEBUG: .../IO/Socket/SSL.pm:718: ssl handshake done
DEBUG: .../IO/Socket/SSL.pm:2588: free ctx 65664688 open=65664688
DEBUG: .../IO/Socket/SSL.pm:2593: free ctx 65664688 callback
DEBUG: .../IO/Socket/SSL.pm:2600: OK free ctx 65664688
500 Status read failed:A non-blocking socket operation could not be completed
immediately
Content-Type: text/plain
Client-Date: Fri, 09 Jan 2015 16:49:31 GMT
Client-Warning: Internal response

Status read failed: A non-blocking socket operation could not be completed immediately
at E:/Perl64/lib/Net/HTTP/Methods.pm line 276.

I guess it's now a little bit different matter, but I searched web, and could not find any answers for this.

Answer 1

I think I see what your problem is. When you do a https proxy request you are not connecting to a proxy with the HTTP protocol, but with the HTTP protocol and then issue a CONNECT request to build the tunnel. So the URL of the proxy must be http:// only and not https://:

$ua->proxy(['https'],'http://199.200.120.140:8089');
                      ^^^^ http:// instead of https://

Apart from that make sure that both LWP::UserAgent and LWP::Protocol::https are at least version 6.06, because these added proper support to proxy HTTPS traffic. But I think the versions should be new enough in your case.

EDIT: since you are running the code on Windows with a recent version of IO::Socket::SSL you've triggered a bug related to the newly added support for non-blocking SSL sockets in Windows, which cause "Status read failed: A non-blocking socket operation could not be completed immediately". For details and a fix see https://github.com/libwww-perl/net-http/pull/11.

Answer 2

Thanks to Steffen, the solution is there https://github.com/libwww-perl/net-http/pull/11 I tried it, it working in my case just fine.