Performance Tuning on Apache 1.3.24 with mod_ssl 2.8.8

Hi, I'd like to know what kind of tricks I can apply
to improve the performance of my apache server which
uses mod_ssl. The OS I'm using is Linux 7.2.

Currently, I have a client script that generates n
number of requests to the apache server. The page it
requests is a static page. With SSL turned on, I'm
only able to get at most 7 to 8 requests per second.
With SSL turned off, I am able to get 50+ requests per
second.

I've tried setting SSLMutex to use sem and
SSLSessionCache to
shm:/usr/local/apache/logs/ssl_gcache_data(512000),
but those changes didn't improve the results.

Any suggestions or ideas? Thanks.

Patrick





__________________________________________________
Do You Yahoo!?
Yahoo! - Official partner of 2002 FIFA World Cup
http://fifaworldcup.yahoo.com
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

Re: Performance Tuning on Apache 1.3.24 with mod_ssl 2.8.8

On Thu, 30 May 2002, Patrick Dionisio wrote:

> Currently, I have a client script that generates n
> number of requests to the apache server. The page it
> requests is a static page. With SSL turned on, I'm
> only able to get at most 7 to 8 requests per second.
> With SSL turned off, I am able to get 50+ requests per
> second.

Wow, that's still incredibly slow. What kind of CPU and how much RAM are
we talking about here? With SSL turned off you should be able to pump out
way more RPS than that on a static page. I suggest you tune that first
(you should be looking for a number in the hundreds of RPS at least), and
*then* focus on SSL. See:

http://httpd.apache.org/docs/misc/perf-tuning.html

Upgrading to Apache 2.0.x might help, too. :)

> I've tried setting SSLMutex to use sem and
> SSLSessionCache to
> shm:/usr/local/apache/logs/ssl_gcache_data(512000),

shmcb can perform better than shmht under stress (shm == shmht in 1.3, shm
== shmcb in 2.0, though you can explicitly specify either choice in both
versions)... that's probably worth looking into. See the thread
http://marc.theaimsgroup.com/?l=apache-modssl&m=985295626294 36&w=2 for an
explanation of the differences (though some of the information there is
out of date by now, eg shmcb is no longer experimental).

> but those changes didn't improve the results.

It should actually be a rather drastic improvement over other session
cache methods. I definitely think you need to concentrate on the rest of
Apache first and then come back to looking at SSL tuning.

--Cliff

____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

RE: Performance Tuning on Apache 1.3.24 with mod_ssl 2.8.8

> -----Original Message-----
> From: Cliff Woolley [mailto:jwoolley@apache.org]
> Sent: 30 May 2002 23:59
> To: modssl-users@modssl.org
> Subject: Re: Performance Tuning on Apache 1.3.24 with mod_ssl 2.8.8
>
>
> On Thu, 30 May 2002, Patrick Dionisio wrote:
>
> > Currently, I have a client script that generates n
> > number of requests to the apache server. The page it
> > requests is a static page. With SSL turned on, I'm
> > only able to get at most 7 to 8 requests per second.
> > With SSL turned off, I am able to get 50+ requests per
> > second.
>
> Wow, that's still incredibly slow. What kind of CPU and how
> much RAM are
> we talking about here? With SSL turned off you should be
> able to pump out
> way more RPS than that on a static page. I suggest you tune
> that first
> (you should be looking for a number in the hundreds of RPS at
> least), and
> *then* focus on SSL. See:
>
> http://httpd.apache.org/docs/misc/perf-tuning.html
>
> Upgrading to Apache 2.0.x might help, too. :)
>
Upgrading to Apache 2.0.x on the users platform (I guess it's Red Hat 7.2)
is particularly hard. I spent a week trying this out recently but kept
running into problems with openssl libraries, and pre-compiled packages.

I used both an rpm that had already been built for Apache 2 (after creating
symlinks to the openssl libraries), and compiled openssl and Apache 2 from
source. In both cases I could send one request for a secure page, but all
subsequent requests hung completely.

Until Red Hat can release an rpm that works with their other rpms I'd
suggest that Apache 2 on that platform is still a bit of a pipe-dream. It's
now my preference to stay with pre-compiled packages where-ever I can,
simply because it is easier for me to administer (but I don't want to start
another discussion on that either!)

Which brings me to the point. Are you using the packages that came with
RedHat 7.2, or compiling your own? In the latter case, you may be seeing
conflicts with the openssl libraries that come with Red Hat 7.2. I've had no
difficulties with the packages that come with Red Hat 7.2 thus far.

-
John Airey
Internet systems support officer, ITCSD, Royal National Institute of the
Blind,
Bakewell Road, Peterborough PE2 6XU,
Tel.: +44 (0) 1733 375299 Fax: +44 (0) 1733 370848 John.Airey@rnib.org.uk

If Charles Darwin knew a fraction of what scientists know today, he'd never
have written the Origin of the Species.

-

NOTICE: The information contained in this email and any attachments is
confidential and may be legally privileged. If you are not the
intended recipient you are hereby notified that you must not use,
disclose, distribute, copy, print or rely on this email's content. If
you are not the intended recipient, please notify the sender
immediately and then delete the email and any attachments from your
system.

RNIB has made strenuous efforts to ensure that emails and any
attachments generated by its staff are free from viruses. However, it
cannot accept any responsibility for any viruses which are
transmitted. We therefore recommend you scan all attachments.

Please note that the statements and views expressed in this email
and any attachments are those of the author and do not necessarily
represent those of RNIB.

RNIB Registered Charity Number: 226227

Website: http://www.rnib.org.uk

14th June 2002 is RNIB Look Loud Day - visit http://www.lookloud.org.uk to
find out all about it.

____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

RE: Performance Tuning on Apache 1.3.24 with mod_ssl 2.8.8

> (but I don't want to start
> another discussion on that either!)
>


Dang! Everyones killing some of my better discussion topics!

Ya'll have a great weekend folks.

Thanks,


Ron DuFresne
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
admin & senior security consultant: sysinfo.com
http://sysinfo.com

"Cutting the space budget really restores my faith in humanity. It
eliminates dreams, goals, and ideals and lets us get straight to the
business of hate, debauchery, and self-annihilation."
-- Johnny Hart

testing, only testing, and damn good at it too!


____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

RE: Performance Tuning on Apache 1.3.24 with mod_ssl 2.8.8

Hi,

generally speaking: encryption of data (which SSL does in comparison to not using SSL) of course cost computing time. Thats the reason why you'll get less processed requests when using SSL. Thats the price for having secure data transfer, which does not mean that you should consider turning off SSL, depending on which site your're running.

Secondly, the results you get from your load test of course strongly depend on it's design, but probably turning on the "KeepAlive" directive may improve your results, depending whether your test script supports this.

NB (I): Is your test client software running on the same server? This would downgrade results, too.
NB (II): A sun Netra T1 (UltraSPARC-IIi 440MHz, Memory 512 MB) (a quite low end server) I recently tested processed about 70 requests per second (using SSL).
NB (II): Which hardware are you're using?


Kind regards,
Bert Courtin


-----Original Message-----
From: Patrick Dionisio [mailto:phdionisio@yahoo.com]
Sent: Thursday, May 30, 2002 10:38 PM
To: modssl-users@modssl.org
Subject: Performance Tuning on Apache 1.3.24 with mod_ssl 2.8.8


Hi, I'd like to know what kind of tricks I can apply
to improve the performance of my apache server which
uses mod_ssl. The OS I'm using is Linux 7.2.

Currently, I have a client script that generates n
number of requests to the apache server. The page it
requests is a static page. With SSL turned on, I'm
only able to get at most 7 to 8 requests per second.
With SSL turned off, I am able to get 50+ requests per
second.

I've tried setting SSLMutex to use sem and
SSLSessionCache to
shm:/usr/local/apache/logs/ssl_gcache_data(512000),
but those changes didn't improve the results.

Any suggestions or ideas? Thanks.

Patrick





__________________________________________________
Do You Yahoo!?
Yahoo! - Official partner of 2002 FIFA World Cup
http://fifaworldcup.yahoo.com
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

Re: Performance Tuning on Apache 1.3.24 with mod_ssl 2.8.8

Hi there,

On Thu, 30 May 2002, Cliff Woolley wrote:

> On Thu, 30 May 2002, Patrick Dionisio wrote:
>
> > Currently, I have a client script that generates n
> > number of requests to the apache server. The page it
> > requests is a static page. With SSL turned on, I'm
> > only able to get at most 7 to 8 requests per second.
> > With SSL turned off, I am able to get 50+ requests per
> > second.
>
> Wow, that's still incredibly slow. What kind of CPU and how much RAM are
> we talking about here? With SSL turned off you should be able to pump out
> way more RPS than that on a static page. I suggest you tune that first
> (you should be looking for a number in the hundreds of RPS at least), and
> *then* focus on SSL. See:

As a first tip - 50 requests per second is very slow already just for
http. I'd look at that first. Don't forget to bear in mind the size of the
page you're pulling down with your http request - multiply that by 50 and
check that you're not approaching any bandwidth limitation of your network
interfaces! :-)

Aside from that - there's a variety of settings in the default apache
config (at least this is true for 1.3.*) that although "generic" and
"helpful" are most certainly not "optimal". Just pulling down
http:/// (ie. the "default page") can involve multiple file
I/O calls by apache just trying to figure out which HTML file to use (ie.
mime-magic, language support, etc). Numerous "Options" directives in
apache slow down generic operation so you may want to wade into the config
file pruning what you can. Likewise, turning off keepalives (which are
evil and should be amputated from all existing and future source) can be a
good idea - the little bit of one-browser-straight-line speed improvement
keepalives give a browser are more than compensated for by the
process-bloat and scalability hassles it gives your server (especially as
modern browsers launch multiple requests in parallel anyway when they want
to "speed up").

I found that I could eek quite a bit of speed improvement out of Apache
just by tweaking the config file and removing fancy (and almost never
used) modules and options.

Then you move onto the ssl-specific stuff ... disabling the "COMPAT" stuff
in mod_ssl is a good idea - last time I checked, the code that populates
environment variables with https-specifics was completely ass-about-face.
I measured something ridiculous like 20,000 strcmp() operations for a
single https handshake. Turning off "compat" support doesn't remove that
problem, but mitigates it somewhat by reducing (substantially) the number
of environment variables modssl tries to populate. Ie. this reduces the
number of iterations of the (slow) loop logic. You also get some mileage
by reducing the verbosity of log output - I'd recommend "Warn" as the
noisiest level you'd want if performance is important (for the regular
Apache LogLevel as well as the modssl-specific one).

> http://httpd.apache.org/docs/misc/perf-tuning.html

You might also want to check the README in the 'swamp' package (shameless
plug, http://www.geoffthorpe.net/crypto/swamp/) - apart from explaining
the usage of 'swamp' (which you may not care for) it does go into a
variety of considerations about client and/or server speeds and how to
meaningfully benchmark and interpret results. Just to start off with,
you've probably (with your https tests) fallen into the first gotcha - EDH
cipher suites. It wouldn't surprise me if your benchmarking program was
negotiating these much slower but higher-security cipher-suites. These
suites aren't actually supported by common browsers anwyay so the
usefulness of those numbers is questionable. OTOH: If you're only getting
50 ops/sec with plain http then it could also just be a hopelessly slow
web server. If it *is* EDH cipher-suites, then your numbers could go up by
a factor of 5 or much more if you test with non-ephemeral suites (eg.
RC4-SHA).

> > I've tried setting SSLMutex to use sem and
> > SSLSessionCache to
> > shm:/usr/local/apache/logs/ssl_gcache_data(512000),
>
> shmcb can perform better than shmht under stress (shm == shmht in 1.3, shm
> == shmcb in 2.0, though you can explicitly specify either choice in both
> versions)... that's probably worth looking into. See the thread
> http://marc.theaimsgroup.com/?l=apache-modssl&m=985295626294 36&w=2 for an
> explanation of the differences (though some of the information there is
> out of date by now, eg shmcb is no longer experimental).

What my failed searches for benchmarking posts *did* turn up was a bit of
info on the 'shmcb' stuff. Eg. some misc. posts of mine that turned up in
that search that touch on session caching and testing (in no particular
order);

(a bit of a monster about 'shmcb')
http://marc.theaimsgroup.com/?l=apache-modssl&m=985310627047 50&w=2

(a bit on swamp usage and session caching)
http://marc.theaimsgroup.com/?l=apache-modssl&m=986511051217 37&w=2

(something else about problems with 'shmht')
http://marc.theaimsgroup.com/?l=apache-modssl&m=999974238022 43&w=2

oh yeah, there's also that security problem with modssl that I mentioned
ages ago - AFAIK this still hasn't been changed in modssl and *may* not
yet have changed in apache 2.0 either. Ralf or David, please correct me
if I'm wrong;

http://marc.theaimsgroup.com/?l=apache-modssl&m=997175851064 20&w=2

> > but those changes didn't improve the results.
>
> It should actually be a rather drastic improvement over other session
> cache methods. I definitely think you need to concentrate on the rest of
> Apache first and then come back to looking at SSL tuning.

Yes, if the move from dbm to shm (any kind of shm for that matter) didn't
show up on your benchmarks, then there's some other kind of sludge in your
setup that is large enough to obscure the benefits of tighter tuning at
the session caching level. Again, I'd recommend taking a look at 'swamp's
README (you can read it online if you don't want to download the package).

Regards,
Geoff

--
Geoff Thorpe, geoff(at)geoffthorpe(dot)net

2000 years on, it's a different empire but the same
zealots and the same attrocities.


____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

Re: Performance Tuning on Apache 1.3.24 with mod_ssl 2.8.8

On Fri, 31 May 2002, Geoff Thorpe wrote:

> oh yeah, there's also that security problem with modssl that I mentioned
> ages ago - AFAIK this still hasn't been changed in modssl and *may* not
> yet have changed in apache 2.0 either. Ralf or David, please correct me
> if I'm wrong;
> http://marc.theaimsgroup.com/?l=apache-modssl&m=997175851064 20&w=2

This was fixed in 2.0 as of 2.0.25 but is not yet fixed in 1.3's modssl.

--Cliff


____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

Re: Performance Tuning on Apache 1.3.24 with mod_ssl 2.8.8

Hi,

On Fri, 31 May 2002, Cliff Woolley wrote:

> On Fri, 31 May 2002, Geoff Thorpe wrote:
>
> > oh yeah, there's also that security problem with modssl that I mentioned
> > ages ago - AFAIK this still hasn't been changed in modssl and *may* not
> > yet have changed in apache 2.0 either. Ralf or David, please correct me
> > if I'm wrong;
> > http://marc.theaimsgroup.com/?l=apache-modssl&m=997175851064 20&w=2
>
> This was fixed in 2.0 as of 2.0.25 but is not yet fixed in 1.3's modssl.

Ah, thanks for the update on that. I mentioned this problem a couple of
times *ages* ago, including private mail to Ralf, but it seemed very few
people seemed to regard it as "an issue". I'm glad Apache 2.0 has taken it
seriously. Ralf, would it be possible to get it similarly incorporated
into the 1.3.* tree? Please?

Cheers,
Geoff


--
Geoff Thorpe, geoff(at)geoffthorpe(dot)net

2000 years on, it's a different empire but the same
zealots and the same attrocities.


____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org