LWP with SSL and Client Side Cookies...

Okay folks, I have a tough one on my hands here.

I have been using LWP::UserAgent for a longtime now. But I need to grab
some data from a fairly sophisticated website. Probably is running JAVA on
the server-side since it rights a JSESSIONID cookie. It actually writes
several cookies, all of which controll authentication and access inside
the site. Its also SSL, but thats not an issue since I have SSLeay
installed.

The cookies are key to being able to get inside the site. I have never
done this with LWP, started playing around with cookie jar - but have not
had any luck.

Since most sites are going in this direction of cookie-authentication,
hopefully some other folks out there have tackled this.

I just need help with the Perl framework that can store and push back that
cookie data during the lifetime of the session - which in my case is the
duration of the perl executable. I have to make about 4 HTTP requests, one
is initial hit to site to load some cookies, 1 is to login and get a few
more cookies!, then 2 more to grab data. I already have all the GET and
POST requests mapped out.

Any ideas?
Thanks
-irwin

RE: LWP with SSL and Client Side Cookies...

intel@quonix.net wrote:
> The cookies are key to being able to get inside the site. I have never
> done this with LWP, started playing around with cookie jar - but have
> not had any luck.

Here's a bit of my custom spider class's new($)...

$self =3D LWP::RobotUA->new(
agent =3D> $_user_agent,
from =3D> $_from,
requests_redirectable =3D>
[ 'GET',
'HEAD',
'POST', # default is just GET and HEAD
],
);

# use cookies
$self->cookie_jar(HTTP::Cookies->new());

I don't need to save cookies from session to session... but if you do, =
there's a way to load a cookie jar from a file.

--=20
Matthew.van.Eerde (at) hbinc.com 805.964.4554 x902
Hispanic Business Inc./HireDiversity.com Software Engineer

RE: LWP with SSL and Client Side Cookies...

I've tried using cookie_jar but I cant seem to get it to work. Here is a
snippet of code that I am using. Its a little dirty, but it hits the
entrance page of the site, grabs the cookies. Where I am stuck is, on my
2nd http request, I need to take those cookies and insert them into the
header of the second request.


use LWP::UserAgent;

$ua = LWP::UserAgent->new(keep_alive => 1);
$ua->agent('Mozilla/5.0')

# Initial non-ssl request to homepage to grab cookies:
#
# JSESSIONID, serverName, and BIGipServerPROD-EXCHANGE-80

my $response_a = $ua->get("http://www.intervalworld.com");

# $response_a->header("set-cookie") now contains my three cookies
# which I can grab with regular expressions.

# Now I need to POST for login and insert those three cookies
# in the request header.

my %formdata = (
login => "abc123",
passwd => "xyz456",
);

# my $response_b = $ua->post("https://secure.intervalworld.com/web/cs", \%formdata )

# Here is where I am stuck, I dont know how to insert
# those previosly stored cookies. I've tried cookie_jar
# but it doesn't seem to work.

-irwin

On Tue, 25 Oct 2005 Matthew.van.Eerde@hbinc.com wrote:

> intel@quonix.net wrote:
> > The cookies are key to being able to get inside the site. I have never
> > done this with LWP, started playing around with cookie jar - but have
> > not had any luck.
>
> Here's a bit of my custom spider class's new($)...
>
> $self = LWP::RobotUA->new(
> agent => $_user_agent,
> from => $_from,
> requests_redirectable =>
> [ 'GET',
> 'HEAD',
> 'POST', # default is just GET and HEAD
> ],
> );
>
> # use cookies
> $self->cookie_jar(HTTP::Cookies->new());
>
> I don't need to save cookies from session to session... but if you do, there's a way to load a cookie jar from a file.
>
> --
> Matthew.van.Eerde (at) hbinc.com 805.964.4554 x902
> Hispanic Business Inc./HireDiversity.com Software Engineer
>
>

Re: LWP with SSL and Client Side Cookies...

On 10/25/05, intel@quonix.net wrote:
> I have been using LWP::UserAgent for a longtime now. But I need to grab
> some data from a fairly sophisticated website. Probably is running JAVA o=
n
> the server-side since it rights a JSESSIONID cookie. It actually writes
> several cookies, all of which controll authentication and access inside
> the site. Its also SSL, but thats not an issue since I have SSLeay
> installed.
>

The same problem that I have... I am struggling since a month on this.
Look at the thread "Can't get authorization".. If you find a way,
please let me know.

Fabio

RE: LWP with SSL and Client Side Cookies...

QUONIX WEB ADMIN wrote:
> I've tried using cookie_jar but I cant seem to get it to work.

Clarify... are you sure it isn't working? Maybe the site is expecting =
something else besides cookies.
> use LWP::UserAgent;
>=20
> $ua =3D LWP::UserAgent->new(keep_alive =3D> 1);
> $ua->agent('Mozilla/5.0')

So at this point you'd add the line
$ua->cookie_jar(HTTP::Cookies->new());

> # Initial non-ssl request to homepage to grab cookies:
> #
> # JSESSIONID, serverName, and BIGipServerPROD-EXCHANGE-80
>=20
> my $response_a =3D $ua->get("http://www.intervalworld.com");

Note the domain... www.intervalworld.com
=20
> # $response_a->header("set-cookie") now contains my three
> cookies # which I can grab with regular expressions.
>=20
> # Now I need to POST for login and insert those three cookies
> # in the request header.
>=20
> my %formdata =3D (
> login =3D> "abc123",
> passwd =3D> "xyz456",
> );
>=20
> # my $response_b =3D
> $ua->post("https://secure.intervalworld.com/web/cs", \%formdata )=20

Ah, the domain is different... www.intervalworld.com ne =
secure.intervalworld.com... so that is why the cookies aren't coming =
across.

All browsers will consider those two URLs to be on "different sites" =
because the server name is different, and won't pass cookies received on =
one to the other.

Maybe your inputs are incorrect? The form that I see shows three inputs =
named "a", "loginID", and "rememberMe".

--=20
Matthew.van.Eerde (at) hbinc.com 805.964.4554 x902
Hispanic Business Inc./HireDiversity.com Software Engineer

Re: LWP with SSL and Client Side Cookies...

I am now getting this error when I try to grab the crucial cookie:

Use of uninitialized value in pattern match (m//) at
/usr/share/perl5/HTTP/Cookies.pm line 45.

RE: LWP with SSL and Client Side Cookies...

Matt,

If that is how cookie_jar behaves, it could explain the problem.

I hit www.intervalworld.com first, get a JSESSIONID cookie, then I hit
secure.intervalworld.com - at which WebSphere wants to see my JSESSIONID
from the previous www.intervalworld.com request.

Though the server name is different, it rights a cookie that is for all
subdomains, i.e. in Netscape cookie manager, site name is listed as
"intervalworld.com", and the cookies domain is listed as
".intervalworld.com".

Maybe LWP's cookie jar is not letting secure.domain read cookie data from
www.domain requests.

However, I need that functionality. If cookie_jar get support it, thats
fine, I have no problem doing it by hand, but I dont know how to manually
insert the cookie data into the request header and the 2nd request.

I have verified that the secure.intervalworld.com POST/GET works - but it
will only work when a JSESSIONID cookie is written from
www.intervalworld.com.

Thanks,
John


On Tue, 25 Oct 2005 Matthew.van.Eerde@hbinc.com wrote:

> QUONIX WEB ADMIN wrote:
> > I've tried using cookie_jar but I cant seem to get it to work.
>
> Clarify... are you sure it isn't working? Maybe the site is expecting something else besides cookies.
> > use LWP::UserAgent;
> >
> > $ua = LWP::UserAgent->new(keep_alive => 1);
> > $ua->agent('Mozilla/5.0')
>
> So at this point you'd add the line
> $ua->cookie_jar(HTTP::Cookies->new());
>
> > # Initial non-ssl request to homepage to grab cookies:
> > #
> > # JSESSIONID, serverName, and BIGipServerPROD-EXCHANGE-80
> >
> > my $response_a = $ua->get("http://www.intervalworld.com");
>
> Note the domain... www.intervalworld.com
>
> > # $response_a->header("set-cookie") now contains my three
> > cookies # which I can grab with regular expressions.
> >
> > # Now I need to POST for login and insert those three cookies
> > # in the request header.
> >
> > my %formdata = (
> > login => "abc123",
> > passwd => "xyz456",
> > );
> >
> > # my $response_b =
> > $ua->post("https://secure.intervalworld.com/web/cs", \%formdata )
>
> Ah, the domain is different... www.intervalworld.com ne secure.intervalworld.com... so that is why the cookies aren't coming across.
>
> All browsers will consider those two URLs to be on "different sites" because the server name is different, and won't pass cookies received on one to the other.
>
> Maybe your inputs are incorrect? The form that I see shows three inputs named "a", "loginID", and "rememberMe".
>
> --
> Matthew.van.Eerde (at) hbinc.com 805.964.4554 x902
> Hispanic Business Inc./HireDiversity.com Software Engineer
>
>

RE: LWP with SSL and Client Side Cookies...

QUONIX WEB ADMIN wrote:
> I hit www.intervalworld.com first, get a JSESSIONID cookie, then I hit
> secure.intervalworld.com - at which WebSphere wants to see my
> JSESSIONID from the previous www.intervalworld.com request.
>=20
> Though the server name is different, it rights a cookie that is for
> all subdomains, i.e. in Netscape cookie manager, site name is listed
> as "intervalworld.com", and the cookies domain is listed as
> ".intervalworld.com".

Hmmm... (tests in Firefox...)
I'm wrong, browsers do allow cross-domain cookies.

--=20
Matthew.van.Eerde (at) hbinc.com 805.964.4554 x902
Hispanic Business Inc./HireDiversity.com Software Engineer