HTTP proxy authentication

Hi,

I have to access internet through a proxy which requires username and passw=
ord.

When I am trying to access a page on internet using WWW::Mechanize
0.33; , I am getting.:

LWP::UserAgent::request: ()
HTTP::Cookies::add_cookie_header: Checking businessx.in for cookies
HTTP::Cookies::add_cookie_header: Checking .in for cookies
LWP::UserAgent::send_request: GET http://www.google.com/
LWP::UserAgent::_need_proxy: Proxied to http://my.proxy:1080
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 734 bytes
LWP::Protocol::collect: read 436 bytes
LWP::Protocol::collect: read 2629 bytes
HTTP::Cookies::extract_cookies: Set cookie BCSI-CS0AD550C8 =3D> 2
LWP::UserAgent::request: Simple response: Proxy Authentication Required

It seems that LWP::UserAgent does not support proxy with authentication.
What is the solution.

TIA,

regards,
Prateek

Re: HTTP proxy authentication

Prateek writes:

> It seems that LWP::UserAgent does not support proxy with authentication.

Proxies with authentication is supported. See:

http://search.cpan.org/~gaas/libwww-perl-5.803/lwpcook.pod#P ROXIES

Regards,
Gisle

Re: HTTP proxy authentication

$ua->proxy()
is not accepting username and password as parameters.

I am having activestate perl on win32.
$ua->env_proxy()
is not helping. The environment variables are correctly set.

I worked around this by using
my $req =3D HTTP::Request->new(GET =3D> $url);
$req->proxy_authorization_basic('username', 'password');
$mech->request( $req);
and this worked fine.

Now, This works well for HTTP but not for HTTPS.
I am having Crypt::SSLeay
But this approach doesn't work.

So, when I try for example $mech->get("https://mail.yahoo.com/");
I again get 407 error.....

It seems that UserAgent works differently for HTTP and HTTPS...
correct me if I am wrong.

regards,
Prateek


On 21 Oct 2005 05:29:42 -0700, Gisle Aas wrote:
> Prateek writes:
>
> > It seems that LWP::UserAgent does not support proxy with authentication=
..
>
> Proxies with authentication is supported. See:
>
> http://search.cpan.org/~gaas/libwww-perl-5.803/lwpcook.pod#P ROXIES
>
> Regards,
> Gisle
>


--
------------------------------------------------------------ -----
Prateek Srivastava
2003 Batch CSE
M M M Engineering College, Gorakhpur.
INDIA

Re: HTTP proxy authentication

Prateek writes:

> $ua->proxy()
> is not accepting username and password as parameters.

Did you read the section pointed to the the URL I provided? It says:

| Some proxies also require that you send it a username/password in
| order to let requests through. You should be able to add the
| required header, with something like this:
|
| use LWP::UserAgent;
|
| $ua = LWP::UserAgent->new;
| $ua->proxy(['http', 'ftp'] => 'http://username:password@proxy.myorg.com');
|
| $req = HTTP::Request->new('GET',"http://www.perl.com");
|
| $res = $ua->request($req);
| print $res->decoded_content if $res->is_success;
|
| Replace proxy.myorg.com, username and password with something
| suitable for your site.


> On 21 Oct 2005 05:29:42 -0700, Gisle Aas wrote:
> > Prateek writes:
> >
> > > It seems that LWP::UserAgent does not support proxy with authentication.
> >
> > Proxies with authentication is supported. See:
> >
> > http://search.cpan.org/~gaas/libwww-perl-5.803/lwpcook.pod#P ROXIES

Re: HTTP proxy authentication

On Fri, Oct 21, 2005 at 12:33:21PM +0530, Prateek (for.groups@gmail.com) wrote:
> When I am trying to access a page on internet using WWW::Mechanize
> 0.33; , I am getting.:

First, your Mechanize is very very old. Get a new one, and take a look
at the Cookbook and FAQ that are now standard.

xoxo,
Andy

--
Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance

Re: HTTP proxy authentication

Thanks ! This worked for HTTP ! I appreciate your quick help.
how to make it work for HTTPS ?
The same code does not work for https://.......

My program has to access http urls and http urls , both..

TIA,
Prateek


On 21 Oct 2005 07:04:58 -0700, Gisle Aas wrote:
> Prateek writes:
>
> > $ua->proxy()
> > is not accepting username and password as parameters.
>
> Did you read the section pointed to the the URL I provided? It says:
>
> | Some proxies also require that you send it a username/password in
> | order to let requests through. You should be able to add the
> | required header, with something like this:
> |
> | use LWP::UserAgent;
> |
> | $ua =3D LWP::UserAgent->new;
> | $ua->proxy(['http', 'ftp'] =3D> 'http://username:password@proxy.myorg.=
com');
> |
> | $req =3D HTTP::Request->new('GET',"http://www.perl.com");
> |
> | $res =3D $ua->request($req);
> | print $res->decoded_content if $res->is_success;
> |
> | Replace proxy.myorg.com, username and password with something
> | suitable for your site.
>
>
> > On 21 Oct 2005 05:29:42 -0700, Gisle Aas wrote:
> > > Prateek writes:
> > >
> > > > It seems that LWP::UserAgent does not support proxy with authentica=
tion.
> > >
> > > Proxies with authentication is supported. See:
> > >
> > > http://search.cpan.org/~gaas/libwww-perl-5.803/lwpcook.pod#P ROXIES
>


--
------------------------------------------------------------ -----
Prateek Srivastava
2003 Batch CSE
M M M Engineering College, Gorakhpur.
INDIA

Re: HTTP proxy authentication

------=_Part_10576_14902251.1129908530101
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Hi,

I am using the following code to access a site as described in your example=
:
(replacing with the values)

#!/usr/bin/perl -w
use strict;
$|++;

use LWP::UserAgent;

my $ua =3D LWP::UserAgent->new;
$ua->proxy(['http', 'ftp', 'https'], 'http://:@<
MyProxy>:1080');

my $req =3D HTTP::Request->new(GET =3D> 'https://www.helsinki.fi/');
my $res =3D $ua->request($req);
if ($res->is_success) {
print $res->as_string;
}
else {
print "Failed: ", $res->status_line, "\n";
}

output is :
Failed: 400 Bad Request

please guide,
TIA,
Prateek


On 10/21/05, Prateek wrote:
> Thanks ! This worked for HTTP ! I appreciate your quick help.
> how to make it work for HTTPS ?
> The same code does not work for https://.......
>
> My program has to access http urls and http urls , both..
>
> TIA,
> Prateek
>
>
> On 21 Oct 2005 07:04:58 -0700, Gisle Aas wrote:
> > Prateek writes:
> >
> > > $ua->proxy()
> > > is not accepting username and password as parameters.
> >
> > Did you read the section pointed to the the URL I provided? It says:
> >
> > | Some proxies also require that you send it a username/password in
> > | order to let requests through. You should be able to add the
> > | required header, with something like this:
> > |
> > | use LWP::UserAgent;
> > |
> > | $ua =3D LWP::UserAgent->new;
> > | $ua->proxy(['http', 'ftp'] =3D> '
http://username:password@proxy.myorg.com');
> > |
> > | $req =3D HTTP::Request->new('GET',"http://www.perl.com");
> > |
> > | $res =3D $ua->request($req);
> > | print $res->decoded_content if $res->is_success;
> > |
> > | Replace proxy.myorg.com , username and
password with something
> > | suitable for your site.
> >
> >
> > > On 21 Oct 2005 05:29:42 -0700, Gisle Aas
wrote:
> > > > Prateek writes:
> > > >
> > > > > It seems that LWP::UserAgent does not support proxy with
authentication.
> > > >
> > > > Proxies with authentication is supported. See:
> > > >
> > > > http://search.cpan.org/~gaas/libwww-perl-5.803/lwpcook.pod#P ROXIES
> >
>
>
> --
> ------------------------------------------------------------ -----
> Prateek Srivastava
> 2003 Batch CSE
> M M M Engineering College, Gorakhpur.
> INDIA
>


--
------------------------------------------------------------ -----
Prateek Srivastava
2003 Batch CSE
M M M Engineering College, Gorakhpur.
INDIA

------=_Part_10576_14902251.1129908530101--

RE: HTTP proxy authentication

Prateek wrote:
> my $ua =3D LWP::UserAgent->new;
> $ua->proxy(['http', 'ftp', 'https'],
> 'http://:@< MyProxy>:1080');
>=20
> output is :
> Failed: 400 Bad Request
>=20
> please guide,

Are you %dd-encoding URI-unsafe characters in your MyUsername and =
MyPassword strings?

--=20
Matthew.van.Eerde (at) hbinc.com 805.964.4554 x902
Hispanic Business Inc./HireDiversity.com Software Engineer

Re: HTTP proxy authentication

------=_Part_10688_6898890.1129909379281
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

I am not doing it but I don't think there is any dangerous character there.
The string is :

'http://r52635:mani-Ratnam@proxy.intra.net:1080'

can this cause a problem?

thanks for the help,

regards,
Prateek


On 10/21/05, Matthew.van.Eerde@hbinc.com
wrote:
>
> Prateek wrote:
> > my $ua =3D LWP::UserAgent->new;
> > $ua->proxy(['http', 'ftp', 'https'],
> > 'http://:@< MyProxy>:1080');
> >
> > output is :
> > Failed: 400 Bad Request
> >
> > please guide,
>
> Are you %dd-encoding URI-unsafe characters in your MyUsername and
> MyPassword strings?
>
> --
> Matthew.van.Eerde (at) hbinc.com 805.964.4554 x902
> Hispanic Business Inc./HireDiversity.com Software Engineer
>



--
------------------------------------------------------------ -----
Prateek Srivastava
2003 Batch CSE
M M M Engineering College, Gorakhpur.
INDIA

------=_Part_10688_6898890.1129909379281--

Re: HTTP proxy authentication

Hi,

I am using the following program to access a HTTPS link through proxy.
Proxy Authentication is required. So, I am specifying the values as
shown below (passwd changed but similar in format):

================================================
#!/usr/bin/perl -w
use strict;
$|++;

use LWP::UserAgent;
use LWP::Debug qw(+);

my $ua = LWP::UserAgent->new;
$ua->proxy(['https'], 'http://x52635:mani-Ratn@proxy.intra.net:1080');

my $req = HTTP::Request->new(GET => 'https://mail.yahoo.com/');
my $res = $ua->request($req);
if ($res->is_success) {
print $res->as_string;
}
else {
print "Failed: ", $res->status_line, "\n";
}
================================================
I am not able to get through. It returns "400 bad Request".
Though, if I only change the link 'https://mail.yahoo.com/' to
'http://mail.yahoo.com/', it gets through.
Please help...
The output of the program is as below:
LWP::UserAgent::new: ()
LWP::UserAgent::proxy: ARRAY(0x225524)
http://x52635:mani-Ratn@proxy.intra.net:1080
LWP::UserAgent::proxy: http
http://x52635:mani-Ratn@proxy.intra.net:1080

LWP::UserAgent::proxy: https
http://x52635:mani-Ratn@proxy.intra.net:1080
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET https://mail.yahoo.com/
LWP::UserAgent::_need_proxy: Proxied to
http://x52635:mani-Ratn@proxy.intra.net:1080
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 862 bytes
LWP::Protocol::collect: read 1716 bytes
LWP::UserAgent::request: Simple response: Bad Request
Failed: 400 Bad Request

Re: HTTP proxy authentication

OK, I did a lot of searching in the archives and tried the following:

================================================
#!/usr/bin/perl -w
use strict;
$|++;

use LWP::UserAgent;
use LWP::Debug qw(+);

my $ua = LWP::UserAgent->new;
$ua->proxy(['http', 'https'],
'http://x52635:mani-Ratn@proxy.intra.net:1080');

my $req = HTTP::Request->new(CONNECT => 'https://mail.yahoo.com/');
my $res = $ua->request($req);
if ($res->is_success) {
print $res->as_string;
}

else {
print "Failed: ", $res->status_line, "\n";
}
================================================
This returns:
################################################
LWP::UserAgent::new: ()
LWP::UserAgent::proxy: ARRAY(0x225524)
http://x52635:mani-Ratn@proxy.intra.net:1080
LWP::UserAgent::proxy: http
http://x52635:mani-Ratn@proxy.intra.net:1080
LWP::UserAgent::proxy: https
http://x52635:mani-Ratn@proxy.intra.net:1080
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: CONNECT https://mail.yahoo.com/
LWP::UserAgent::_need_proxy: Proxied to
http://x52635:mani-Ratn@proxy.intra.net:1080
LWP::Protocol::http::request: ()
LWP::UserAgent::request: Simple response: OK
HTTP/1.1 200 (OK) Connection established
Client-Date: Sun, 23 Oct 2005 10:49:23 GMT
Client-Peer: 10.213.80.200:1080
Proxy-Agent: BlueCoat-Security-Appliance
################################################
But, I do not get any content. If only I change CONNECT to GET,
I get:
################################################
LWP::UserAgent::new: ()
LWP::UserAgent::proxy: ARRAY(0x225524)
http://x52635:mani-Ratn@proxy.intra.net:1080
LWP::UserAgent::proxy: http
http://x52635:mani-Ratn@proxy.intra.net:1080
LWP::UserAgent::proxy: https
http://x52635:mani-Ratn@proxy.intra.net:1080
LWP::UserAgent::request: ()
LWP::UserAgent::send_request: GET https://mail.yahoo.com/
LWP::UserAgent::_need_proxy: Proxied to
http://x52635:mani-Ratn@proxy.intra.net:1080
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 862 bytes
LWP::Protocol::collect: read 1716 bytes
LWP::UserAgent::request: Simple response: Bad Request
Failed: 400 Bad Request
################################################
So, the culprit was I was that CONNECT was not being sent.
But, how do I get the content?