Javascript and WWW::Mechanize or LWP::UserAgent?

Hello,

I've been using WWW::Mechanize to automate testing of cgi scripts -
works great!

The problem I'm running into is "trying to deal with scripts that use
javascript" - so far, I've had to ignore them or, at least, those
widgets/forms that rely on javascript. Googling hasn't led anywhere
besides statements that Mech doesn't handle javascript - I got that far
myself :-)

Is there a way to get Mech to handle javascript? Has anyone been able
to do this? Anyone working on it? Or have I missed something?

I really like being able to test from my unix box instead of having to
find a windows box to run quicktest pro or winrunner on.

Thanks,

Warren

Re: Javascript and WWW::Mechanize or LWP::UserAgent?

On Mon, Jul 11, 2005 at 05:13:04PM -0400, Warren Pollans (wpollans@gmail.com) wrote:
> Is there a way to get Mech to handle javascript? Has anyone been able
> to do this? Anyone working on it? Or have I missed something?

No. No. Theoretically. No.

--
Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance

RE: Javascript and WWW::Mechanize or LWP::UserAgent?

What's the theoretical reason?=20

-----Original Message-----
From: Andy Lester [mailto:andy@petdance.com]=20
Sent: Monday, July 11, 2005 2:14 PM
To: Warren Pollans
Cc: libwww@perl.org
Subject: Re: Javascript and WWW::Mechanize or LWP::UserAgent?

On Mon, Jul 11, 2005 at 05:13:04PM -0400, Warren Pollans =
(wpollans@gmail.com) wrote:
> Is there a way to get Mech to handle javascript? Has anyone been able =

> to do this? Anyone working on it? Or have I missed something?

No. No. Theoretically. No.

--
Andy Lester =3D> andy@petdance.com =3D> www.petdance.com =3D> =
AIM:petdance

Re: Javascript and WWW::Mechanize or LWP::UserAgent?

Last year I was trying to extend some of Sean Burke's DOM stuff with a
primitive pure perl JavaScript parser and classes to mimic JS objects.
It's difficult and I didn't get far before running out of time and
putting it aside. Probably the right avenue is this sort of approach,
http://search.cpan.org/dist/JavaScript/JavaScript.pod (riding on top of
one of the two open source JS engines) + an HTML DOM parser.

It would be sweet and widely beneficial if it ever happened (Google
might even pay for it if it isn't too late for the summer of code
thing). I enjoy playing with this stuff, and still would be if I had
time, but it's a little out of my league. I humbly entreat one of my
betters to give it a go.

-Ashley

On Monday, July 11, 2005, at 02:13 PM, Andy Lester wrote:

> On Mon, Jul 11, 2005 at 05:13:04PM -0400, Warren Pollans
> (wpollans@gmail.com) wrote:
>> Is there a way to get Mech to handle javascript? Has anyone been able
>> to do this? Anyone working on it? Or have I missed something?
>
> No. No. Theoretically. No.
>
> --
> Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance
>
>

Re: Javascript and WWW::Mechanize or LWP::UserAgent?

On Mon, Jul 11, 2005 at 02:18:32PM -0700, Lan, Fang (flan@amazon.com) wrote:
> What's the theoretical reason?

I know of someone who told me last week that he was working on a
JavaScript implementation. I haven't heard anything since.

--
Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance

RE: Javascript and WWW::Mechanize or LWP::UserAgent?

I think the "theoretically" is the answer to the "Is anyone working on
this?" question.

Someone may be working on it, but it may be a while before the work is
done. A long while.

That's mainly a guess, hope it helps.

Jim

On Jul 11 Lan, Fang wrote:

> What's the theoretical reason?
>
> -----Original Message-----
> From: Andy Lester [mailto:andy@petdance.com]
> Sent: Monday, July 11, 2005 2:14 PM
> To: Warren Pollans
> Cc: libwww@perl.org
> Subject: Re: Javascript and WWW::Mechanize or LWP::UserAgent?
>
> On Mon, Jul 11, 2005 at 05:13:04PM -0400, Warren Pollans (wpollans@gmail.com) wrote:
> > Is there a way to get Mech to handle javascript? Has anyone been able
> > to do this? Anyone working on it? Or have I missed something?
>
> No. No. Theoretically. No.
>
> --
> Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance
>

Re: Javascript and WWW::Mechanize or LWP::UserAgent?

--------------040200050602070505070406
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi Warren,

I wrote some comments on dealing with javascript from a scraper for this
list. You can find the comments, e.g. at
http://www.codecomments.com/message443297.html - It doesn't sound like
that's really a solution to your problem, however, since presumably you
want to test the javascript produced by the cgi.

An other alternative, if you are using Windows - there is a iex-based
implementation of Mech, *Win32::IE::Mechanize*

which uses Internet explorer instead of libwww as the underlying browser
implementation. I have no idea in what state the code is or whether
javascript works, but it might be worth looking into (and I'd really
appreciate hearing some feedback on how it works, because I have this
problem, too ;-) ).

Cheers,

Peter



Warren Pollans wrote:

> Hello,
>
> I've been using WWW::Mechanize to automate testing of cgi scripts -
> works great!
>
> The problem I'm running into is "trying to deal with scripts that use
> javascript" - so far, I've had to ignore them or, at least, those
> widgets/forms that rely on javascript. Googling hasn't led anywhere
> besides statements that Mech doesn't handle javascript - I got that
> far myself :-)
>
> Is there a way to get Mech to handle javascript? Has anyone been able
> to do this? Anyone working on it? Or have I missed something?
>
> I really like being able to test from my unix box instead of having to
> find a windows box to run quicktest pro or winrunner on.
>
> Thanks,
>
> Warren
>
>

--------------040200050602070505070406--

Re: Javascript and WWW::Mechanize or LWP::UserAgent?

On Mon, 11 Jul 2005, Warren Pollans wrote:

> I've been using WWW::Mechanize to automate testing of cgi scripts -
> works great!
>
> The problem I'm running into is "trying to deal with scripts that use
> javascript" - so far, I've had to ignore them or, at least, those
[...]
> I really like being able to test from my unix box instead of having to
> find a windows box to run quicktest pro or winrunner on.

Try Selenium instead of mechanize. Not ideal for scraping (you have to
drag in the entire browser, including its GUI, to use it, and the "driven"
mode is currently buggy), but good for functional testing. Written in
cross-browser JavaScript (in good OO style, too).

http://selenium.thoughtworks.com/


It's quite new, but it's the only free tool in its class (cross-browser
playback of functional tests) that I know of. With any luck, somebody may
write a decent test recorder for it too (ie. record a test simply by doing
what you'd do if you were manually testing -- ATM there are HTTP-proxying
test recorders that try and do this, but one written in JS is what's
really needed).


John

Re: Javascript and WWW::Mechanize or LWP::UserAgent?

--------------060300000703030008000706
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

You might also try writing your own javascript parsing routines?

my $mech = WWW::Mechanize->new();
$mech->get("blah.html");

use some method in some way

if (my $result = do_something($mech)){#do something}


then in some method

sub do_something {
my $mech = shift;
my $blah;
my $stream = HTML::TokeParser->new(\$mech->{content});
#use tokeparser or even treebuilder

return $blah;
}



Or maybe you could add javascript parsing routines to WWW::Mechanize and
submit a patch?

I don't know enough about your particular problem....


John J Lee wrote:

>On Mon, 11 Jul 2005, Warren Pollans wrote:
>
>
>
>>I've been using WWW::Mechanize to automate testing of cgi scripts -
>>works great!
>>
>>The problem I'm running into is "trying to deal with scripts that use
>>javascript" - so far, I've had to ignore them or, at least, those
>>
>>
>[...]
>
>
>>I really like being able to test from my unix box instead of having to
>>find a windows box to run quicktest pro or winrunner on.
>>
>>
>
>Try Selenium instead of mechanize. Not ideal for scraping (you have to
>drag in the entire browser, including its GUI, to use it, and the "driven"
>mode is currently buggy), but good for functional testing. Written in
>cross-browser JavaScript (in good OO style, too).
>
>http://selenium.thoughtworks.com/
>
>
>It's quite new, but it's the only free tool in its class (cross-browser
>playback of functional tests) that I know of. With any luck, somebody may
>write a decent test recorder for it too (ie. record a test simply by doing
>what you'd do if you were manually testing -- ATM there are HTTP-proxying
>test recorders that try and do this, but one written in JS is what's
>really needed).
>
>
>John
>
>
>
>

--------------060300000703030008000706--

Re: Javascript and WWW::Mechanize or LWP::UserAgent?

[Warren Pollans]
> The problem I'm running into is "trying to deal with scripts that use
> javascript" - so far, I've had to ignore them or, at least, those

[deborah sciales]
> You might also try writing your own javascript parsing routines?

That's not a small amount of work you've just set Warren to do. :-)

(speaking as somebody who made a semi-serious attempt at it, in Python)


John

Re: Javascript and WWW::Mechanize or LWP::UserAgent?

--------------080704060201080008060304
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

Well, I guess it depends on his set of needs, and he does have
tokeparser and treebuilder, etc to use.

If his javascript is inside of script tags, he can use treebuilder to
get those nodes, and then work with them. I see a host of Javascript
modules on CPAN.

Here's why I would not try to write this in Python just yet, unless i
had the time:

Perl -MCPAN -e shell;

cpan> i/JavaScript/


use a module from CPAN, use a few modules from CPAN, or patch and
improve a module!

The reason we are still using Perl for fast work, or a major reason is
CPAN. LWP is a part of that.

I don't have time to look through the javascript modules, but something
might be useful, and you could incorporate it into your current test
script base without switching to a new testing enviro...?


John J Lee wrote:

>[Warren Pollans]
>
>
>>The problem I'm running into is "trying to deal with scripts that use
>>javascript" - so far, I've had to ignore them or, at least, those
>>
>>
>
>[deborah sciales]
>
>
>>You might also try writing your own javascript parsing routines?
>>
>>
>
>That's not a small amount of work you've just set Warren to do. :-)
>
>(speaking as somebody who made a semi-serious attempt at it, in Python)
>
>
>John
>
>
>
>

--------------080704060201080008060304--

LWP::UserAgent and TCP_NODELAY

Hello, I have a LWP application that POSTs thousands of small files to a
mod_perl server. All POSTs *under* a certain size (around 3k) seem to
take about 200x as long as those over. In this case about 0.09 seconds
as opposed to 0.004 seconds.

After much debugging I narrowed the issue down to TCP latency and
discovered I can get around the problem by setting TCP_NODELAY on the
socket.

My workaround is redefining the LWP::Protocol::http::_check_sock()
method and adding a setsockopt($sock, 6, 1, pack("l", 1)). This tweak
significantly improves performance on small POSTs but feels like a total
hack.

Can I be sure that the _check_sock() method will always be there? The
function itself doesn't do anything so I assume its just a hook. Is
there another, better way to do this? Thanks much.

mike

Re: Javascript and WWW::Mechanize or LWP::UserAgent?

--------------020107010901010008020104
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Under the heading of small serious amounts of work...

I mentioned previously Win32::IE::Mechanize - does anybody have any
ideas on how to do the same thing with Firefox under Linux?

Cheers,
Peter

John J Lee wrote:

>[Warren Pollans]
>
>
>>The problem I'm running into is "trying to deal with scripts that use
>>javascript" - so far, I've had to ignore them or, at least, those
>>
>>
>
>[deborah sciales]
>
>
>>You might also try writing your own javascript parsing routines?
>>
>>
>
>That's not a small amount of work you've just set Warren to do. :-)
>
>(speaking as somebody who made a semi-serious attempt at it, in Python)
>
>
>John
>
>
>
>

--------------020107010901010008020104--

Re: Javascript and WWW::Mechanize or LWP::UserAgent?

[John Lee]
> That's not a small amount of work you've just set Warren to do. :-)
>
> (speaking as somebody who made a semi-serious attempt at it, in Python)

[deborah sciales]
> Well, I guess it depends on his set of needs, and he does have
> tokeparser and treebuilder, etc to use.
>
> If his javascript is inside of script tags, he can use treebuilder to
> get those nodes, and then work with them. I see a host of Javascript
> modules on CPAN.
>
> Here's why I would not try to write this in Python just yet, unless i
> had the time:
>
> Perl -MCPAN -e shell;
>
> cpan> i/JavaScript/
>
>
> use a module from CPAN, use a few modules from CPAN, or patch and
> improve a module!

I don't think people here are interested in Python/Perl comparisons.

Nevertheless, the problems with the existing libraries for either language
are roughly the same, AFAICT (I too started with existing libraries.
That was kind of the easy part).

Are there a *specific* set of HTML parsing, HTML (not XML) DOM-building
(with all the