some flawed benchmarks

am 10.07.2008 07:28:18 von Adam Prime

A couple of months ago i was going through slides from gozers "From =20
CGI to mod_perl 2.0, Fast!" talk, which has some benchmarks comparing =20
CGI, perlrun and registry to each other. At which point i realized =20
that i've never really known how much faster using straight handlers =20
is than using one of the CGI emulation layers. I also didn't have any =20
idea how much faster SetHandler modperl was vs SetHandler perl-script.

So i decided to see what i could figure out. I took gozers CGI from =20
the slides (slightly modified) and ran it through the paces on my =20
laptop, then converted the script to run as a straight handler.

here's the CGI version:

#!/usr/bin/perl

print qq[Content-Type: text/html\r\n\r\n];

print(qq[

Hello Worlds



GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}

MOD_PERL: $ENV{MOD_PERL}

]);

Here's the Handler version

package Kabob::HelloWorld;

use strict;
use warnings;

use Apache2::RequestRec ();

use Apache2::Const -compile =3D>qw(:common);

sub handler {
my $r =3D shift;

$r->content_type('text/html');
$r->print(qq[

Hello Worlds



GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}

MOD_PERL: $ENV{MOD_PERL}

]);

return Apache2::Const::OK;
}

1;

and here's the conf (these tests were all running through a light =20
mod_proxy front end too)

ProxyPass http://localhost:8080/cgi/
ProxyPassReverse http://localhost:8080/cgi/

ScriptAlias /cgi/ /www/p/

Alias /perlrun/ /www/p/

ProxyPass http://localhost:8080/perlrun/
ProxyPassReverse http://localhost:8080/perlrun/

SetHandler perl-script
PerlHandler ModPerl::PerlRun
Options +ExecCGI
PerlSendHeader On

Alias /registry/ /www/p/

ProxyPass http://localhost:8080/registry/
ProxyPassReverse http://localhost:8080/registry/

SetHandler perl-script
PerlHandler ModPerl::Registry
Options +ExecCGI
PerlSendHeader On

ProxyPass http://localhost:8080/perlscript/
ProxyPassReverse http://localhost:8080/perlscript/

SetHandler perl-script
PerlResponseHandler Kabob::HelloWorld

ProxyPass http://localhost:8080/modperl/
ProxyPassReverse http://localhost:8080/modperl/

SetHandler modperl
PerlResponseHandler Kabob::HelloWorld

and here's the results (which are no doubt flawed for a number of reasons)

running: ab -n 10000 [url]

CGI
Requests per second: 217.80 [#/sec] (mean)
Time per request: 4.591 [ms] (mean)
Transfer rate: 53.17 [Kbytes/sec] received

PerlRun
Requests per second: 482.49 [#/sec] (mean)
Time per request: 2.073 [ms] (mean)
Transfer rate: 114.49 [Kbytes/sec] received

Registry
Requests per second: 693.33 [#/sec] (mean)
Time per request: 1.442 [ms] (mean)
Transfer rate: 164.53 [Kbytes/sec] received

SetHandler perl-script
Requests per second: 772.12 [#/sec] (mean)
Time per request: 1.295 [ms] (mean)
Transfer rate: 189.94 [Kbytes/sec] received

SetHandler modperl
Requests per second: 1048.66 [#/sec] (mean)
Time per request: 0.954 [ms] (mean)
Transfer rate: 250.84 [Kbytes/sec] received

I'm not sure how well you can really compare the CGI emulation numbers =20
to the PerlHandler numbers, but personally i think the 30%ish =20
improvement from perl-script to modperl is pretty amazing. I wouldn't =20
have imagined it would have been that high.

Adam

Re: some flawed benchmarks

am 10.07.2008 11:39:36 von el.dodgero

I appreciate this, as I'd been wondering.

But it also prompts me to.. I gotta ask...

No offence but...
Don't you know what a here_doc is?

--
Dodger

2008/7/9 :
> A couple of months ago i was going through slides from gozers "From CGI to
> mod_perl 2.0, Fast!" talk, which has some benchmarks comparing CGI, perlrun
> and registry to each other. At which point i realized that i've never
> really known how much faster using straight handlers is than using one of
> the CGI emulation layers. I also didn't have any idea how much faster
> SetHandler modperl was vs SetHandler perl-script.
>
> So i decided to see what i could figure out. I took gozers CGI from the
> slides (slightly modified) and ran it through the paces on my laptop, then
> converted the script to run as a straight handler.
>
> here's the CGI version:
>
> #!/usr/bin/perl
>
>
> print qq[Content-Type: text/html\r\n\r\n];
>
> print(qq[
>
>

Hello Worlds



> GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}

> MOD_PERL: $ENV{MOD_PERL}

>

>
>
> ]);
>
>
> Here's the Handler version
>
> package Kabob::HelloWorld;
>
> use strict;
> use warnings;
>
> use Apache2::RequestRec ();
>
> use Apache2::Const -compile =>qw(:common);
>
> sub handler {
> my $r = shift;
>
> $r->content_type('text/html');
> $r->print(qq[
>
>

Hello Worlds



> GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}

> MOD_PERL: $ENV{MOD_PERL}

>

>
>
>
> ]);
>
> return Apache2::Const::OK;
> }
>
> 1;
>
> and here's the conf (these tests were all running through a light mod_proxy
> front end too)
>
>
>
>
> ProxyPass http://localhost:8080/cgi/
> ProxyPassReverse http://localhost:8080/cgi/
>
>
>
> ScriptAlias /cgi/ /www/p/
>
>
>
> Alias /perlrun/ /www/p/
>
>
> ProxyPass http://localhost:8080/perlrun/
> ProxyPassReverse http://localhost:8080/perlrun/
>
>
> SetHandler perl-script
> PerlHandler ModPerl::PerlRun
> Options +ExecCGI
> PerlSendHeader On
>
>
>
> Alias /registry/ /www/p/
>
>
> ProxyPass http://localhost:8080/registry/
> ProxyPassReverse http://localhost:8080/registry/
>
>
> SetHandler perl-script
> PerlHandler ModPerl::Registry
> Options +ExecCGI
> PerlSendHeader On
>
>
>
>
>
>
>
> ProxyPass http://localhost:8080/perlscript/
> ProxyPassReverse http://localhost:8080/perlscript/
>
>
> SetHandler perl-script
> PerlResponseHandler Kabob::HelloWorld
>
>
>
>
>
> ProxyPass http://localhost:8080/modperl/
> ProxyPassReverse http://localhost:8080/modperl/
>
>
> SetHandler modperl
> PerlResponseHandler Kabob::HelloWorld
>
>
>
>
> and here's the results (which are no doubt flawed for a number of reasons)
>
> running: ab -n 10000 [url]
>
> CGI
> Requests per second: 217.80 [#/sec] (mean)
> Time per request: 4.591 [ms] (mean)
> Transfer rate: 53.17 [Kbytes/sec] received
>
> PerlRun
> Requests per second: 482.49 [#/sec] (mean)
> Time per request: 2.073 [ms] (mean)
> Transfer rate: 114.49 [Kbytes/sec] received
>
> Registry
> Requests per second: 693.33 [#/sec] (mean)
> Time per request: 1.442 [ms] (mean)
> Transfer rate: 164.53 [Kbytes/sec] received
>
> SetHandler perl-script
> Requests per second: 772.12 [#/sec] (mean)
> Time per request: 1.295 [ms] (mean)
> Transfer rate: 189.94 [Kbytes/sec] received
>
> SetHandler modperl
> Requests per second: 1048.66 [#/sec] (mean)
> Time per request: 0.954 [ms] (mean)
> Transfer rate: 250.84 [Kbytes/sec] received
>
> I'm not sure how well you can really compare the CGI emulation numbers to
> the PerlHandler numbers, but personally i think the 30%ish improvement from
> perl-script to modperl is pretty amazing. I wouldn't have imagined it would
> have been that high.
>
> Adam
>
>
>
>
>
>
>
>
>

--
Dodger

Re: some flawed benchmarks

am 10.07.2008 11:47:28 von el.dodgero

Oh. I would also recommend three variants, based on what people often
do, what people sometimes do, and what people probably should do when
using CGI.pm, which can make a difference (just for thoroughness):

Usually done:
#!/usr/bin/perl
use CGI;
print header;

print <<"EOF";

Environment dump:

$_: $ENV{$_}

EOF

Sometimes do:
#!/usr/bin/perl
use strict;
use CGI;
my $cgi = new CGI;
print $cgi->header;

print <<"EOF";

Environment dump:

$_: $ENV{$_}

EOF

Might do occassionally, and probably should do all the time if using CGI:

#!/usr/bin/perl
use strict;
use CGI(); # note the difference -- using CGI in OO mode, don't import
*anything*
my $cgi = new CGI;
print $cgi->header;

print <<"EOF";

Environment dump:

$_: $ENV{$_}

EOF

2008/7/9 :
> A couple of months ago i was going through slides from gozers "From CGI to
> mod_perl 2.0, Fast!" talk, which has some benchmarks comparing CGI, perlrun
> and registry to each other. At which point i realized that i've never
> really known how much faster using straight handlers is than using one of
> the CGI emulation layers. I also didn't have any idea how much faster
> SetHandler modperl was vs SetHandler perl-script.
>
> So i decided to see what i could figure out. I took gozers CGI from the
> slides (slightly modified) and ran it through the paces on my laptop, then
> converted the script to run as a straight handler.
>
> here's the CGI version:
>
> #!/usr/bin/perl
>
>
> print qq[Content-Type: text/html\r\n\r\n];
>
> print(qq[
>
>

Hello Worlds



> GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}

> MOD_PERL: $ENV{MOD_PERL}

>

Hello Worlds



> GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}

> MOD_PERL: $ENV{MOD_PERL}

>

Re: some flawed benchmarks

am 10.07.2008 11:55:02 von Brock Diegel

On 10 Jul 2008, at 06:28, adam.prime@utoronto.ca wrote:
> I'm not sure how well you can really compare the CGI emulation
> numbers to the PerlHandler numbers, but personally i think the
> 30%ish improvement from perl-script to modperl is pretty amazing. I
> wouldn't have imagined it would have been that high.

It would be interesting to see how FCGI compares to those numbers.

--
Andy Armstrong, Hexten

Re: some flawed benchmarks

am 10.07.2008 14:50:01 von Adam Prime

I deliberately removed CGI from the script because i personally would
never use CGI in something written to be run as straight handlers, and
it obviously wouldn't make any sense to use CGI in the CGI emulations,
and then not use it in the Handler version.

Not using heredoc's shouldn't really have any effect on the (already
flawed) results, because they weren't used in any of the examples. But
yeah, it was late and i should have been sleeping but i was dorking
around with this instead.

I'm personally not really interested in how using CGI affects the
numbers, but if other people would like to see them i can do this later
tonight.

Adam

Dodger wrote:
> Oh. I would also recommend three variants, based on what people often
> do, what people sometimes do, and what people probably should do when
> using CGI.pm, which can make a difference (just for thoroughness):
>
> Usually done:
> #!/usr/bin/perl
> use CGI;
> print header;
>
> print <<"EOF";
>
>
>

Environment dump:

$_: $ENV{$_}

>
>
> EOF
>
> Sometimes do:
> #!/usr/bin/perl
> use strict;
> use CGI;
> my $cgi = new CGI;
> print $cgi->header;
>
> print <<"EOF";
>
>
>

Environment dump:

$_: $ENV{$_}

>
>
> EOF
>
> Might do occassionally, and probably should do all the time if using CGI:
>
> #!/usr/bin/perl
> use strict;
> use CGI(); # note the difference -- using CGI in OO mode, don't import
> *anything*
> my $cgi = new CGI;
> print $cgi->header;
>
> print <<"EOF";
>
>
>

Environment dump:

$_: $ENV{$_}

>
>
> EOF
>

Re: some flawed benchmarks

am 10.07.2008 14:51:23 von Adam Prime

Andy Armstrong wrote:
> It would be interesting to see how FCGI compares to those numbers.

I don't know anything about fastcgi, but i suppose i could look at that
also this evening.

Adam

Re: some flawed benchmarks

am 10.07.2008 15:26:57 von Perrin Harkins

On Thu, Jul 10, 2008 at 1:28 AM, wrote:
> and here's the conf (these tests were all running through a light mod_proxy
> front end too)

Note that CGI and FastCGI don't need the proxy frontend.

> I'm not sure how well you can really compare the CGI emulation numbers to
> the PerlHandler numbers, but personally i think the 30%ish improvement from
> perl-script to modperl is pretty amazing. I wouldn't have imagined it would
> have been that high.

It is pretty cool that it works so well. I feel like I should point
out though, for the benefit of those using Registry, that the
difference is only this big because the code isn't doing anything. In
a real-world scenario you'd see an improvement, but nothing close to
30%.

- Perrin

Re: some flawed benchmarks

am 10.07.2008 16:05:13 von Adam Prime

Quoting Perrin Harkins :

> Note that CGI and FastCGI don't need the proxy frontend.

The only reason I did it that way was because that's how apache was =20
already set up on my laptop, and i didn't feel like dorking around =20
with it too much. I could certainly change it around so that CGI and =20
FGCGI (if added) were handled by the light apache. I wasn't sure how =20
apples to oranges that would be, but i guess in reality, people that =20
are running plain CGI or fast CGI probably aren't doing it on a =20
mod_perl enabled server.

> It is pretty cool that it works so well. I feel like I should point
> out though, for the benefit of those using Registry, that the
> difference is only this big because the code isn't doing anything. In
> a real-world scenario you'd see an improvement, but nothing close to
> 30%.

It's probably also worth noting that under sethandler modperl the =20
GATEWAY_INTERFACE env variable has no value, which results in the page =20
being 3 bytes shorter (iirc), which is actually a pretty big deal =20
since the whole page is pretty short. The examples should likely be =20
modified to actually return pages that are the same length.

anything else stupid i'm missing?

adam

Re: some flawed benchmarks

am 11.07.2008 05:43:49 von Adam Prime

i've changed some stuff and added fastcgi to the mix. i've given =20
numbers through the proxy as well as without it for all the mod_perl =20
examples

I've modified the scripts so that they producing the same amount of =20
output for each example (except cgi, which adds a content-length =20
header, so the total data transfered is larger)

here's the cgi, used under cgi, perlrun and registry:

#!/usr/bin/perl

use strict;

print qq[Content-Type: text/html\r\n\r\n];

my $blurb =3D qq[1234567890\n] x 100;

print <

Hello Worlds



$blurb

EOF

here's the fastcgi version:

#!/usr/bin/perl

use strict;
use FCGI;

while (FCGI::accept >=3D 0) {

print qq[Content-Type: text/html\r\n\r\n];

my $blurb =3D qq[1234567890\n] x 100;

print <

Hello Worlds



$blurb

EOF

}

and here's the handler, used for perlscript and modperl:

package Kabob::HelloWorld;

use strict;

use Apache2::RequestRec ();
use Apache2::Const -compile =3D>qw(:common);

sub handler {
my $r =3D shift;

$r->content_type('text/html');

my $blurb =3D qq[1234567890\n] x 100;

$r->print(<

Hello Worlds



$blurb

EOF

return Apache2::Const::OK;
}

1;

This is the conf entry for fast-cgi

Alias /fcgi/ /www/p/

SetHandler fastcgi-script
Options +ExecCGI

the conf entries for the other examples remained the same as they were =20
in the original email. Obviously i didn't get to the different =20
methods of CGI that dodger suggested. maybe over the weekend.

results ordered by transfer rate:

cgi - no proxy
Total transferred: 13080000 bytes
HTML transferred: 11640000 bytes
Requests per second: 95.58 [#/sec] (mean)
Time per request: 10.462 [ms] (mean)
Transfer rate: 122.09 [Kbytes/sec] received

perlrun - through proxy
Total transferred: 12860000 bytes
HTML transferred: 11640000 bytes
Requests per second: 463.07 [#/sec] (mean)
Time per request: 2.160 [ms] (mean)
Transfer rate: 581.52 [Kbytes/sec] received

registry - through proxy
Total transferred: 12860000 bytes
HTML transferred: 11640000 bytes
Requests per second: 641.11 [#/sec] (mean)
Time per request: 1.560 [ms] (mean)
Transfer rate: 805.10 [Kbytes/sec] received

perlrun - no proxy
Total transferred: 12860000 bytes
HTML transferred: 11640000 bytes
Requests per second: 658.17 [#/sec] (mean)
Time per request: 1.519 [ms] (mean)
Transfer rate: 826.53 [Kbytes/sec] received

perlscript - through proxy
Total transferred: 12860000 bytes
HTML transferred: 11640000 bytes
Requests per second: 788.59 [#/sec] (mean)
Time per request: 1.268 [ms] (mean)
Transfer rate: 990.31 [Kbytes/sec] received

modperl - through proxy
Total transferred: 12860000 bytes
HTML transferred: 11640000 bytes
Requests per second: 1063.74 [#/sec] (mean)
Time per request: 0.940 [ms] (mean)
Transfer rate: 1335.84 [Kbytes/sec] received

registry - no proxy
Total transferred: 12860000 bytes
HTML transferred: 11640000 bytes
Requests per second: 1160.28 [#/sec] (mean)
Time per request: 0.862 [ms] (mean)
Transfer rate: 1457.08 [Kbytes/sec] received

fastcgi - no proxy
Total transferred: 12860000 bytes
HTML transferred: 11640000 bytes
Requests per second: 1201.28 [#/sec] (mean)
Time per request: 0.832 [ms] (mean)
Transfer rate: 1508.57 [Kbytes/sec] received

perlscript - no proxy
Total transferred: 12860000 bytes
HTML transferred: 11640000 bytes
Requests per second: 1430.31 [#/sec] (mean)
Time per request: 0.699 [ms] (mean)
Transfer rate: 1796.18 [Kbytes/sec] received

modperl - no proxy
Total transferred: 12860000 bytes
HTML transferred: 11640000 bytes
Requests per second: 2430.80 [#/sec] (mean)
Time per request: 0.411 [ms] (mean)
Transfer rate: 3052.59 [Kbytes/sec] received

With the proxy taken out of the mix, the jump from perlscript to =20
modperl is even more pronounced.

If anyone's got any ideas about why the proxy is having as significant =20
an impact as it is, i'd love to tweak it. Also worth noting is that =20
the front end is running worker, the backend is running prefork.

adam

Re: some flawed benchmarks

am 11.07.2008 06:12:53 von Perrin Harkins

On Thu, Jul 10, 2008 at 11:43 PM, wrote:
> If anyone's got any ideas about why the proxy is having as significant an
> impact as it is, i'd love to tweak it.

There are some proxy settings in recent mod_proxy versions (apache
2.2) which you could experiment with. I haven't tried them yet. I
think you can use persistent connections to the backend with this
setup, since it can pool them with the worker MPM.

You could also experiment with the config on the mod_perl server. For
the most part, if you run with concurrency below MaxClients and you do
a warmup run before each benchmark, it should mitigate common config
issues.

> Also worth noting is that the front
> end is running worker, the backend is running prefork.

That's a good setup. Some people use lighttpd, perlbal, nginx, etc.
for proxying. If you feel ambitious you could try one of those.

- Perrin