Re: https / proxy problem (LWP::UserAgent)

"Larry" said:
>I have a script on Windows which uses LWP to make a simple GET request
>through a proxy:
>
> #!/usr/bin/perl
>
> use strict;
> use warnings;
> use LWP::UserAgent;
>
> my $url = "https://mysite.com";
>
> my $ua = LWP::UserAgent->new;
> $ua->proxy(['http', 'https'] => 'http://myproxy.org:80');
> my $response = $ua->get( $url );
>
> $response->is_success or
> die "Failed to GET '$url': ", $response->status_line;
>
> print $response->as_string, "\n";
>
>I also have Crypt::SSLeay installed so that I can access https sites.
>When I try to run the script as above, it prints some HTTP headers and
>an HTML-formatted error message from the firewall server which says
>"Scheme not supported".

Could it be that LWP does not support HTTP 'CONNECT' method for working
with proxies? As it is this what is needed to proxy https requests:
you first make a 'CONNECT' request to the proxy, and then place your
https traffic in the connection that the proxy opened for you.
--
Wolf a.k.a. Juha Laiho Espoo, Finland
(GC 3.0) GIT d- s+: a C++ ULSH++++$ P++@ L+++ E- W+$@ N++ !K w !O !M V
PS(+) PE Y+ PGP(+) t- 5 !X R !tv b+ !DI D G e+ h---- r+++ y++++
"...cancel my subscription to the resurrection!" (Jim Morrison)

Re: https / proxy problem (LWP::UserAgent)

Juha Laiho wrote:
> "Larry" said:
> >I have a script on Windows which uses LWP to make a simple GET request
> >through a proxy:
> >
> > #!/usr/bin/perl
> >
> > use strict;
> > use warnings;
> > use LWP::UserAgent;
> >
> > my $url = "https://mysite.com";
> >
> > my $ua = LWP::UserAgent->new;
> > $ua->proxy(['http', 'https'] => 'http://myproxy.org:80');
> > my $response = $ua->get( $url );
> >
> > $response->is_success or
> > die "Failed to GET '$url': ", $response->status_line;
> >
> > print $response->as_string, "\n";
> >
> >I also have Crypt::SSLeay installed so that I can access https sites.
> >When I try to run the script as above, it prints some HTTP headers and
> >an HTML-formatted error message from the firewall server which says
> >"Scheme not supported".
>
> Could it be that LWP does not support HTTP 'CONNECT' method for working
> with proxies? As it is this what is needed to proxy https requests:
> you first make a 'CONNECT' request to the proxy, and then place your
> https traffic in the connection that the proxy opened for you.

I took out the "get" call and replaced it with CONNECT as follows:

$ua->proxy(['http', 'https'] => 'http://myproxy.org:80');
my $request = new HTTP::Request CONNECT => $url;
my $response = $ua->request($request);
print $response->as_string;

and it printed:

HTTP/1.0 200 (OK)
Client-Date: Fri, 13 Jan 2006 18:28:05 GMT
Client-Peer: 10.10.15.22:80

but if I change the CONNECT to:

my $request = new HTTP::Request GET => $url;

I get:

HTTP/1.0 200 (OK)
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 90
Content-Type: text/html
Client-Date: Fri, 13 Jan 2006 18:33:20 GMT
Client-Peer: 10.10.15.22:80
Client-Response-Num: 1
Title: Error

Error

Re: https / proxy problem (LWP::UserAgent)

Juha Laiho wrote:
> "Larry" said:
> >I have a script on Windows which uses LWP to make a simple GET request
> >through a proxy:
> >
> > #!/usr/bin/perl
> >
> > use strict;
> > use warnings;
> > use LWP::UserAgent;
> >
> > my $url = "https://mysite.com";
> >
> > my $ua = LWP::UserAgent->new;
> > $ua->proxy(['http', 'https'] => 'http://myproxy.org:80');
> > my $response = $ua->get( $url );
> >
> > $response->is_success or
> > die "Failed to GET '$url': ", $response->status_line;
> >
> > print $response->as_string, "\n";
> >
> >I also have Crypt::SSLeay installed so that I can access https sites.
> >When I try to run the script as above, it prints some HTTP headers and
> >an HTML-formatted error message from the firewall server which says
> >"Scheme not supported".
>
> Could it be that LWP does not support HTTP 'CONNECT' method for working
> with proxies? As it is this what is needed to proxy https requests:
> you first make a 'CONNECT' request to the proxy, and then place your
> https traffic in the connection that the proxy opened for you.

I took out the "get" call and replaced it with CONNECT as follows:

$ua->proxy(['http', 'https'] => 'http://myproxy.org:80');
my $request = new HTTP::Request CONNECT => $url;
my $response = $ua->request($request);
print $response->as_string;

and it printed:

HTTP/1.0 200 (OK)
Client-Date: Fri, 13 Jan 2006 18:28:05 GMT
Client-Peer: 10.10.15.22:80

but if I change the CONNECT to:

my $request = new HTTP::Request GET => $url;

I get:

HTTP/1.0 200 (OK)
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 90
Content-Type: text/html
Client-Date: Fri, 13 Jan 2006 18:33:20 GMT
Client-Peer: 10.10.15.22:80
Client-Response-Num: 1
Title: Error

Error

Re: https / proxy problem (LWP::UserAgent)

"Larry" said:
>Juha Laiho wrote:
>> "Larry" said:
>> >I have a script on Windows which uses LWP to make a simple GET request
>> >through a proxy:
....
>> >I also have Crypt::SSLeay installed so that I can access https sites.
>> >When I try to run the script as above, it prints some HTTP headers and
>> >an HTML-formatted error message from the firewall server which says
>> >"Scheme not supported".
>>
>> Could it be that LWP does not support HTTP 'CONNECT' method for working
>> with proxies? As it is this what is needed to proxy https requests:
>> you first make a 'CONNECT' request to the proxy, and then place your
>> https traffic in the connection that the proxy opened for you.
>
>I took out the "get" call and replaced it with CONNECT as follows:

It wouldn't be quite that simple; 'CONNECT' is something that would leave
you with a connection to the proxy, and would instruct your proxy to
open a connection to the remote machine. Using the same connection
you should then start your SSL traffic.

However, I did dig a little bit deeper into Google with this issue, and
found the following discussion which appears to have a solution for you:
http://groups.google.fi/group/perl.libwww/browse_thread/thre ad/aee0e06b1404387/e28d1e3670a21ad5?lnk=st&q=%22https+throug h+proxy+with+libwww%22&rnum=1#e28d1e3670a21ad5

In short:
- with LWP and Crypt::SSLeay you should be able to do as documented in
Crypt::SSLeay documentation - that is, set environment variable
HTTPS_PROXY, and not set any proxy directives for LWP
(and after this in your perl code you just do 'GET' as you would
do without any proxy in between)
- with LWP and IO::Socket::SSL looks like accessing SSL sites through
a proxy is not possible (I'm happy to accept corrections for this)
- without LWP, HTTPS through proxy appears to be possible with just
Net::SSLeay
--
Wolf a.k.a. Juha Laiho Espoo, Finland
(GC 3.0) GIT d- s+: a C++ ULSH++++$ P++@ L+++ E- W+$@ N++ !K w !O !M V
PS(+) PE Y+ PGP(+) t- 5 !X R !tv b+ !DI D G e+ h---- r+++ y++++
"...cancel my subscription to the resurrection!" (Jim Morrison)

Re: https / proxy problem (LWP::UserAgent)

"Larry" said:
>Juha Laiho wrote:
>> "Larry" said:
>> >I have a script on Windows which uses LWP to make a simple GET request
>> >through a proxy:
....
>> >I also have Crypt::SSLeay installed so that I can access https sites.
>> >When I try to run the script as above, it prints some HTTP headers and
>> >an HTML-formatted error message from the firewall server which says
>> >"Scheme not supported".
>>
>> Could it be that LWP does not support HTTP 'CONNECT' method for working
>> with proxies? As it is this what is needed to proxy https requests:
>> you first make a 'CONNECT' request to the proxy, and then place your
>> https traffic in the connection that the proxy opened for you.
>
>I took out the "get" call and replaced it with CONNECT as follows:

It wouldn't be quite that simple; 'CONNECT' is something that would leave
you with a connection to the proxy, and would instruct your proxy to
open a connection to the remote machine. Using the same connection
you should then start your SSL traffic.

However, I did dig a little bit deeper into Google with this issue, and
found the following discussion which appears to have a solution for you:
http://groups.google.fi/group/perl.libwww/browse_thread/thre ad/aee0e06b1404387/e28d1e3670a21ad5?lnk=st&q=%22https+throug h+proxy+with+libwww%22&rnum=1#e28d1e3670a21ad5

In short:
- with LWP and Crypt::SSLeay you should be able to do as documented in
Crypt::SSLeay documentation - that is, set environment variable
HTTPS_PROXY, and not set any proxy directives for LWP
(and after this in your perl code you just do 'GET' as you would
do without any proxy in between)
- with LWP and IO::Socket::SSL looks like accessing SSL sites through
a proxy is not possible (I'm happy to accept corrections for this)
- without LWP, HTTPS through proxy appears to be possible with just
Net::SSLeay
--
Wolf a.k.a. Juha Laiho Espoo, Finland
(GC 3.0) GIT d- s+: a C++ ULSH++++$ P++@ L+++ E- W+$@ N++ !K w !O !M V
PS(+) PE Y+ PGP(+) t- 5 !X R !tv b+ !DI D G e+ h---- r+++ y++++
"...cancel my subscription to the resurrection!" (Jim Morrison)

Re: https / proxy problem (LWP::UserAgent)

Juha Laiho wrote:
> "Larry" said:
> >Juha Laiho wrote:
> >> "Larry" said:
> >> >I have a script on Windows which uses LWP to make a simple GET request
> >> >through a proxy:
> ...
> >> >I also have Crypt::SSLeay installed so that I can access https sites.
> >> >When I try to run the script as above, it prints some HTTP headers and
> >> >an HTML-formatted error message from the firewall server which says
> >> >"Scheme not supported".
> >>
>
> However, I did dig a little bit deeper into Google with this issue, and
> found the following discussion which appears to have a solution for you:
> http://groups.google.fi/group/perl.libwww/browse_thread/thre ad/aee0e06b1404387/e28d1e3670a21ad5?lnk=st&q=%22https+throug h+proxy+with+libwww%22&rnum=1#e28d1e3670a21ad5
>
> In short:
> - with LWP and Crypt::SSLeay you should be able to do as documented in
> Crypt::SSLeay documentation - that is, set environment variable
> HTTPS_PROXY, and not set any proxy directives for LWP
> (and after this in your perl code you just do 'GET' as you would
> do without any proxy in between)
> - with LWP and IO::Socket::SSL looks like accessing SSL sites through
> a proxy is not possible (I'm happy to accept corrections for this)
> - without LWP, HTTPS through proxy appears to be possible with just
> Net::SSLeay

I tried the first method you suggested and it worked perfectly! Thanks
so much... this is just what we needed! We were really out of ideas,
and this helped a whole lot.

Re: https / proxy problem (LWP::UserAgent)

Juha Laiho wrote:
> "Larry" said:
> >Juha Laiho wrote:
> >> "Larry" said:
> >> >I have a script on Windows which uses LWP to make a simple GET request
> >> >through a proxy:
> ...
> >> >I also have Crypt::SSLeay installed so that I can access https sites.
> >> >When I try to run the script as above, it prints some HTTP headers and
> >> >an HTML-formatted error message from the firewall server which says
> >> >"Scheme not supported".
> >>
>
> However, I did dig a little bit deeper into Google with this issue, and
> found the following discussion which appears to have a solution for you:
> http://groups.google.fi/group/perl.libwww/browse_thread/thre ad/aee0e06b1404387/e28d1e3670a21ad5?lnk=st&q=%22https+throug h+proxy+with+libwww%22&rnum=1#e28d1e3670a21ad5
>
> In short:
> - with LWP and Crypt::SSLeay you should be able to do as documented in
> Crypt::SSLeay documentation - that is, set environment variable
> HTTPS_PROXY, and not set any proxy directives for LWP
> (and after this in your perl code you just do 'GET' as you would
> do without any proxy in between)
> - with LWP and IO::Socket::SSL looks like accessing SSL sites through
> a proxy is not possible (I'm happy to accept corrections for this)
> - without LWP, HTTPS through proxy appears to be possible with just
> Net::SSLeay

I tried the first method you suggested and it worked perfectly! Thanks
so much... this is just what we needed! We were really out of ideas,
and this helped a whole lot.