https through proxy with libwww

hello libwww@perl.org !

I need to access https pages and work with forms there. I tried like
that:

use strict;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new;
$ua->agent('Mozilla/5.0');
$ua->proxy(['https', 'http', 'ftp'] => 'http://my.proxy.addr:3128');
my $req = HTTP::Request->new(GET => 'https://some.secured.host/');
my $res = $ua->request($req);
if ($res->is_success) {
print $res->as_string;
}
else {
print "Failed: ", $res->status_line, "\n";
}

but it gives me
Failed: 501 Not Implemented
This piece of code works just fine without proxy defined, but i need
proxy. As far as i understood, https through proxy is not implemented
yet. Maybe you guys know another way to get it working with https
through proxy?

---
best regards
Evgeny Stepanov

Re: https through proxy with libwww

On Fri, Mar 18, 2005 at 06:39:38PM +0300, Evgeny Stepanov wrote:
> use strict;
> use LWP::UserAgent;
>
> my $ua = LWP::UserAgent->new;
> $ua->agent('Mozilla/5.0');
> $ua->proxy(['https', 'http', 'ftp'] => 'http://my.proxy.addr:3128');
> my $req = HTTP::Request->new(GET => 'https://some.secured.host/');
> my $res = $ua->request($req);
> if ($res->is_success) {
> print $res->as_string;
> }
> else {
> print "Failed: ", $res->status_line, "\n";
> }
>
> but it gives me
> Failed: 501 Not Implemented
> This piece of code works just fine without proxy defined, but i need
> proxy. As far as i understood, https through proxy is not implemented
> yet. Maybe you guys know another way to get it working with https
> through proxy?

Evgeny,

What you are trying to do is to initiate a 'GET' while most proxies
would expect method 'CONNECT'. With a GET the proxy would have to open
an SSL connection. With an end-to-end technology like SSL this is
pretty senseless. How should a proxy check the server cert or how
should the proxy send a client cert?

Sometimes it helps to read the docs. In Crypt::SSLeay it says

Crypt::SSLeay Proxy Support

For native Crypt::SSLeay proxy support of https requests, you
need to set an environment variable HTTPS_PROXY to your proxy
server & port, as in:

# PROXY SUPPORT
$ENV{HTTPS_PROXY} = 'http://proxy_hostname_or_ip:port';
$ENV{HTTPS_PROXY} = '127.0.0.1:8080';

ride or break the Crypt::SSLeay support, so do not mix the two.
Use of the HTTPS_PROXY environment variable in this way is
similar to LWP::UserAgent->env_proxy() usage, but calling that
method will likely over

And, voila, in the proxy logs you should see

==> /var/log/squid/access.log <==
1111195481.685 656 192.168..X.XX TCP_MISS/200 8934 CONNECT some.secured.host:443 - DIRECT/XXXXXXXXX -

\rho

Re: R: https through proxy with libwww

Hello Andrea,

Friday, March 18, 2005, 8:34:01 PM, âû íàïèñàëè:

AS> Have you tried using NET::SSLeay module within the script?
no, i have Crypt::SSLeay, it was written in the docs that LWP would
use this package. In the same readme was mentioned that LWP could use
NET::SSLeay, but it has no direct bindings for LWP. Whatever that
means, i don't know much of perl, so i have no idea how to use
NET::SSLeay in my script. Maybe some piece of example code will help.
Thanks in advance!

---
best regards
Evgeny Stepanov

Re[2]: https through proxy with libwww

Hello Robert,

RB> Sometimes it helps to read the docs. In Crypt::SSLeay it says
yes, i've read LWP docs but not Crypt::SSLeay :-) Thanks for pointing
me to the right place!
RB> Crypt::SSLeay Proxy Support
RB> For native Crypt::SSLeay proxy support of https requests, you
RB> need to set an environment variable HTTPS_PROXY to your proxy
RB> server & port, as in:
RB> # PROXY SUPPORT
RB> $ENV{HTTPS_PROXY} = 'http://proxy_hostname_or_ip:port';
wow! that works! Thanks, robert!

---
best regards
Evgeny Stepanov

Strange HTTP::Headers behaviour in WWW::Mechanize->{res}->{_headers} hash

Hi List.

I have an issue to submit to you:

When i loop through WWW::Mechanize->{res}->{_headers} like the following
snippet:

my $headers = $mech->{res}->{_headers};
while ( (my $key, my $value) = each (%$headers)) {
print $key . ': ' . "$value\n";}
}


I get some values as an array:
For instance the 'content-type' key, the 'links' key and the 'expires' key
are seen as an array, with one or more values each (almost) identical.
In example, i have 'content-type' key with two values: [0] =
'text/html;charset=iso-8859-1' and [1] = ' text/html;charset=iso-8859-1'

Please note the space at the beginning of value [1].

How does it happens? It happens when i load a php self-referencing page for
Tutos planner login, on other sites it behaves correctly (without any
hash-of-array).

Thank you all for helping,

Andrea Setti.

Re: Strange HTTP::Headers behaviour in WWW::Mechanize->{res}->{_headers} hash

On Tue, Mar 22, 2005 at 10:41:10AM +0100, Andrea Setti wrote:
> Hi List.
>
> I have an issue to submit to you:
>
> When i loop through WWW::Mechanize->{res}->{_headers} like the following
> snippet:
>
> my $headers = $mech->{res}->{_headers};
> while ( (my $key, my $value) = each (%$headers)) {
> print $key . ': ' . "$value\n";}
> }

Did you check whether there are actually API calls to do what
you need to do? If I see a component _something in a hash, this
usually tells me to keep my fingers off.

\rho

R: Strange HTTP::Headers behaviour in WWW::Mechanize->{res}->{_headers} hash

Yes, i know, but i didn't found any API that gets the headers from a =
page
and forward them to the browser...

I'm trying to make a single sign-on script, and i need to do the =
following
steps:

1. fetch the original login page and submit username & password
2. submit the login page and fetch from the HTTP::Headers all the =
cookies
that i need to forward to the browser
3. forward to the browser all the cookies / cache informations / other
headers and the content of the authenticated page.

The problem is that i am not able to forward a full HTTP::Message to the
browser, it does not work (ie: $my_HTMessage->as_string();)
So, the only way i have found to forward the cookies to the browser is =
to
get them from the $mech->(res)->_headers, and print them in stdout =
building
my own headers followed by the authenticated content...

if anyone has a better idea, please let me know...


Thank you for supporting!


-----Messaggio originale-----
Da: Robert Barta [mailto:rho@mando] Per conto di Robert Barta
Inviato: gioved=EC 24 marzo 2005 03.49
A: Andrea Setti
Cc: libwww@perl.org; 'John J Lee'
Oggetto: Re: Strange HTTP::Headers behaviour in
WWW::Mechanize->{res}->{_headers} hash

On Tue, Mar 22, 2005 at 10:41:10AM +0100, Andrea Setti wrote:
> Hi List.
>=20
> I have an issue to submit to you:
>=20
> When i loop through WWW::Mechanize->{res}->{_headers} like the=20
> following
> snippet:
>=20
> my $headers =3D $mech->{res}->{_headers}; while ( (my $key, my $value) =
=
> each (%$headers)) {
> print $key . ': ' . "$value\n";}
> }

Did you check whether there are actually API calls to do what you need =
to
do? If I see a component _something in a hash, this usually tells me to =
keep
my fingers off.

\rho

Re: R: Strange HTTP::Headers behaviour in WWW::Mechanize->{res}->{_headers} hash

On Thu, Mar 24, 2005 at 09:31:08AM +0100, Andrea Setti wrote:
> Yes, i know, but i didn't found any API that gets the headers from a page
> and forward them to the browser...
>
> I'm trying to make a single sign-on script, and i need to do the following
> steps:
>
> 1. fetch the original login page and submit username & password
> 2. submit the login page and fetch from the HTTP::Headers all the cookies
> that i need to forward to the browser
> 3. forward to the browser all the cookies / cache informations / other
> headers and the content of the authenticated page.

If it is only the cookie stuff then reading

man lwpcook

may help. Remember, that WWW::Mechanize subclasses LWP.

Otherwise I am not sure what headers WHICH THE SERVER sends. you
should put into a message WHICH then THE CLIENT SENDS. There are not
too many occasions where you would do that...

\rho

> -----Messaggio originale-----
> Da: Robert Barta [mailto:rho@mando] Per conto di Robert Barta
> Inviato: gioved? 24 marzo 2005 03.49
> A: Andrea Setti
> Cc: libwww@perl.org; 'John J Lee'
> Oggetto: Re: Strange HTTP::Headers behaviour in
> WWW::Mechanize->{res}->{_headers} hash
>
> On Tue, Mar 22, 2005 at 10:41:10AM +0100, Andrea Setti wrote:
> > Hi List.
> >
> > I have an issue to submit to you:
> >
> > When i loop through WWW::Mechanize->{res}->{_headers} like the
> > following
> > snippet:
> >
> > my $headers = $mech->{res}->{_headers}; while ( (my $key, my $value) =
> > each (%$headers)) {
> > print $key . ': ' . "$value\n";}
> > }
>
> Did you check whether there are actually API calls to do what you need to
> do? If I see a component _something in a hash, this usually tells me to keep
> my fingers off.
>
> \rho
>

R: R: Strange HTTP::Headers behaviour in WWW::Mechanize->{res}->{_headers} hash

> If it is only the cookie stuff then reading=20
>
> man lwpcook
>
> may help. Remember, that WWW::Mechanize subclasses LWP.

I have created my mech obj as this: my $mech =3D
WWW::Mechanize->new(cookie_jar =3D> {});
This would create an empty cookie jar that can be used to fetch cookie
infos.
But it is not so convenient when you need also other parameters from the
headers, i think it could be easier to build a brand new HTTP::Header =
with
all the needed infos...

> Otherwise I am not sure what headers WHICH THE SERVER sends. you =
should
put into a message WHICH then THE CLIENT SENDS. There are not too many
occasions > where you would do that...

Well, for instance, i need to fetch all the "link" header, that informs =
the
bowser to use a certain css stuff, links etc...
Maybe would be necessary to fetch also the cache-control header, the
transfer-encoding...



What i would like to create is a "fake" login page, with no need of
submitting username and password. This could be very useful in a portal
enviromnent...


=20

-----Messaggio originale-----
Da: Robert Barta [mailto:rho@mando] Per conto di Robert Barta
Inviato: gioved=EC 24 marzo 2005 10.27
A: Andrea Setti
Cc: libwww@perl.org
Oggetto: Re: R: Strange HTTP::Headers behaviour in
WWW::Mechanize->{res}->{_headers} hash

On Thu, Mar 24, 2005 at 09:31:08AM +0100, Andrea Setti wrote:
> Yes, i know, but i didn't found any API that gets the headers from a=20
> page and forward them to the browser...
>=20
> I'm trying to make a single sign-on script, and i need to do the=20
> following
> steps:
>=20
> 1. fetch the original login page and submit username & password 2.=20
> submit the login page and fetch from the HTTP::Headers all the cookies =

> that i need to forward to the browser 3. forward to the browser all=20
> the cookies / cache informations / other headers and the content of=20
> the authenticated page.

If it is only the cookie stuff then reading=20

man lwpcook

may help. Remember, that WWW::Mechanize subclasses LWP.

Otherwise I am not sure what headers WHICH THE SERVER sends. you should =
put
into a message WHICH then THE CLIENT SENDS. There are not too many =
occasions
where you would do that...

\rho

> -----Messaggio originale-----
> Da: Robert Barta [mailto:rho@mando] Per conto di Robert Barta
> Inviato: gioved? 24 marzo 2005 03.49
> A: Andrea Setti
> Cc: libwww@perl.org; 'John J Lee'
> Oggetto: Re: Strange HTTP::Headers behaviour in=20
> WWW::Mechanize->{res}->{_headers} hash
>=20
> On Tue, Mar 22, 2005 at 10:41:10AM +0100, Andrea Setti wrote:
> > Hi List.
> >=20
> > I have an issue to submit to you:
> >=20
> > When i loop through WWW::Mechanize->{res}->{_headers} like the=20
> > following
> > snippet:
> >=20
> > my $headers =3D $mech->{res}->{_headers}; while ( (my $key, my =
$value)=20
> > =3D each (%$headers)) {
> > print $key . ': ' . "$value\n";}
> > }
>=20
> Did you check whether there are actually API calls to do what you need =

> to do? If I see a component _something in a hash, this usually tells=20
> me to keep my fingers off.
>=20
> \rho
>=20

Re: R: R: Strange HTTP::Headers behaviour in WWW::Mechanize->{res}->{_headers} hash

On Thu, Mar 24, 2005 at 11:07:51AM +0100, Andrea Setti wrote:
> I have created my mech obj as this: my $mech =
> WWW::Mechanize->new(cookie_jar => {});
> This would create an empty cookie jar that can be used to fetch cookie
> infos.
> But it is not so convenient when you need also other parameters from the
> headers, i think it could be easier to build a brand new HTTP::Header with
> all the needed infos...
>
> > Otherwise I am not sure what headers WHICH THE SERVER sends. you should
> put into a message WHICH then THE CLIENT SENDS. There are not too many
> occasions > where you would do that...
>
> Well, for instance, i need to fetch all the "link" header, that informs the
> bowser to use a certain css stuff, links etc...

You talk about



headers? These are HTML headers, NOT HTTP headers.

> What i would like to create is a "fake" login page, with no need of
> submitting username and password. This could be very useful in a portal
> enviromnent...

Hmmm.

\rho

> -----Messaggio originale-----
> Da: Robert Barta [mailto:rho@mando] Per conto di Robert Barta
> Inviato: gioved? 24 marzo 2005 10.27
> A: Andrea Setti
> Cc: libwww@perl.org
> Oggetto: Re: R: Strange HTTP::Headers behaviour in
> WWW::Mechanize->{res}->{_headers} hash
>
> On Thu, Mar 24, 2005 at 09:31:08AM +0100, Andrea Setti wrote:
> > Yes, i know, but i didn't found any API that gets the headers from a
> > page and forward them to the browser...
> >
> > I'm trying to make a single sign-on script, and i need to do the
> > following
> > steps:
> >
> > 1. fetch the original login page and submit username & password 2.
> > submit the login page and fetch from the HTTP::Headers all the cookies
> > that i need to forward to the browser 3. forward to the browser all
> > the cookies / cache informations / other headers and the content of
> > the authenticated page.
>
> If it is only the cookie stuff then reading
>
> man lwpcook
>
> may help. Remember, that WWW::Mechanize subclasses LWP.
>
> Otherwise I am not sure what headers WHICH THE SERVER sends. you should put
> into a message WHICH then THE CLIENT SENDS. There are not too many occasions
> where you would do that...
>
> \rho
>
> > -----Messaggio originale-----
> > Da: Robert Barta [mailto:rho@mando] Per conto di Robert Barta
> > Inviato: gioved? 24 marzo 2005 03.49
> > A: Andrea Setti
> > Cc: libwww@perl.org; 'John J Lee'
> > Oggetto: Re: Strange HTTP::Headers behaviour in
> > WWW::Mechanize->{res}->{_headers} hash
> >
> > On Tue, Mar 22, 2005 at 10:41:10AM +0100, Andrea Setti wrote:
> > > Hi List.
> > >
> > > I have an issue to submit to you:
> > >
> > > When i loop through WWW::Mechanize->{res}->{_headers} like the
> > > following
> > > snippet:
> > >
> > > my $headers = $mech->{res}->{_headers}; while ( (my $key, my $value)
> > > = each (%$headers)) {
> > > print $key . ': ' . "$value\n";}
> > > }
> >
> > Did you check whether there are actually API calls to do what you need
> > to do? If I see a component _something in a hash, this usually tells
> > me to keep my fingers off.
> >
> > \rho
> >
>