CRL Checking Uses Excessive Memory

I work for the DoD. We have about a dozen CA's with their own CRL files.
Some of these are over 20M in size. When CRL checking is enabled in Apache
(for Linux or Windows), memory use is excessive and httpd processes are
killed by the OS (Linux) due to out of memory conditions and all the memory
swapping activity sends the proc utilization way up there and makes the
server unresponsive. On Windows the CPU use just pegs at 100% (I have no
idea what else is going on in there).
CRL's are downloaded every day and openssl is used to make hash'd file names
(ssl.conf is using SSLCARevocationPath). I don't currently restart apache
after retrieving the new CRL files.
The Linux machine runs redhat with dual 3ghz xeons and 2Gb ram. SSL works
great, but as soon as CRLs are checked, apache starts to go south! I have a
2Gb swap partition and have added another 2Gb swap file to at least keep
things running, but it becomes so slow it might as well crash.
Each httpd process goes from using about 14Mb of memory when not CRL
checking to 250Mb when CRL checking is enabled!
BTW: anywhere from 10 to 20 concurrent httpd processes are normal for that
machine.

Any ideas on how to use large CRL's in Apache?

Do I just need more memory?

If Apache can't use many large CRL files, would an OSCP solution side-step
these problems? Any good ones out there?
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

Re: CRL Checking Uses Excessive Memory

I think the first thing you need to do is connect to this URL
from someplace that doesn't have any certs related to you
installed, like your local library:

https://www.hill.af.mil/main/index.html

I am not trying to be funny, I am just worried that either you
are going to get yourself into trouble by exposing configuration
info about .mil computers, or somebody else is going to get into
trouble while trying to help you.

Phil

Walls Rob W Contr 75 CS/SCBS wrote:
> I work for the DoD. We have about a dozen CA's with their own CRL files.
> Some of these are over 20M in size. When CRL checking is enabled in Apache
> (for Linux or Windows), memory use is excessive and httpd processes are
> killed by the OS (Linux) due to out of memory conditions and all the memory
> swapping activity sends the proc utilization way up there and makes the
> server unresponsive. On Windows the CPU use just pegs at 100% (I have no
> idea what else is going on in there).
> CRL's are downloaded every day and openssl is used to make hash'd file names
> (ssl.conf is using SSLCARevocationPath). I don't currently restart apache
> after retrieving the new CRL files.
> The Linux machine runs redhat with dual 3ghz xeons and 2Gb ram. SSL works
> great, but as soon as CRLs are checked, apache starts to go south! I have a
> 2Gb swap partition and have added another 2Gb swap file to at least keep
> things running, but it becomes so slow it might as well crash.
> Each httpd process goes from using about 14Mb of memory when not CRL
> checking to 250Mb when CRL checking is enabled!
> BTW: anywhere from 10 to 20 concurrent httpd processes are normal for that
> machine.
>
> Any ideas on how to use large CRL's in Apache?
>
> Do I just need more memory?
>
> If Apache can't use many large CRL files, would an OSCP solution side-step
> these problems? Any good ones out there?
> ____________________________________________________________ __________
> Apache Interface to OpenSSL (mod_ssl) www.modssl.org
> User Support Mailing List modssl-users@modssl.org
> Automated List Manager majordomo@modssl.org

--
Phil Ehrens | Fun stuff:
The LIGO Laboratory, MS 18-34 | http://www.ralphmag.org
California Institute of Technology | http://www.yellow5.com
1200 East California Blvd. | http://www.tokyotosho.com
Pasadena, CA 91125 USA | My gpg public key:
Phone:(626)395-8518 Fax:(626)793-9744 | http://www.imbe.net/peligo.asc
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

RE: CRL Checking Uses Excessive Memory

Hi Phil,

As far as I know, nothing that Rob mentioned is classified...especially
since he is not naming systems by name or address. The fact that the DoD
uses certificates is no secret...there's been many writeups in the various
trade magazines regarding the DoD's push to PKI.

Dwight...

-----Original Message-----
From: owner-modssl-users@modssl.org
[mailto:owner-modssl-users@modssl.org]On Behalf Of Phil Ehrens
Sent: Friday, April 21, 2006 11:11 AM
To: modssl-users@modssl.org
Subject: Re: CRL Checking Uses Excessive Memory


I think the first thing you need to do is connect to this URL
from someplace that doesn't have any certs related to you
installed, like your local library:

https://www.hill.af.mil/main/index.html

I am not trying to be funny, I am just worried that either you
are going to get yourself into trouble by exposing configuration
info about .mil computers, or somebody else is going to get into
trouble while trying to help you.

Phil

Walls Rob W Contr 75 CS/SCBS wrote:
> I work for the DoD. We have about a dozen CA's with their own CRL files.
> Some of these are over 20M in size. When CRL checking is enabled in Apache
> (for Linux or Windows), memory use is excessive and httpd processes are
> killed by the OS (Linux) due to out of memory conditions and all the
memory
> swapping activity sends the proc utilization way up there and makes the
> server unresponsive. On Windows the CPU use just pegs at 100% (I have no
> idea what else is going on in there).
> CRL's are downloaded every day and openssl is used to make hash'd file
names
> (ssl.conf is using SSLCARevocationPath). I don't currently restart apache
> after retrieving the new CRL files.
> The Linux machine runs redhat with dual 3ghz xeons and 2Gb ram. SSL works
> great, but as soon as CRLs are checked, apache starts to go south! I have
a
> 2Gb swap partition and have added another 2Gb swap file to at least keep
> things running, but it becomes so slow it might as well crash.
> Each httpd process goes from using about 14Mb of memory when not CRL
> checking to 250Mb when CRL checking is enabled!
> BTW: anywhere from 10 to 20 concurrent httpd processes are normal for that
> machine.
>
> Any ideas on how to use large CRL's in Apache?
>
> Do I just need more memory?
>
> If Apache can't use many large CRL files, would an OSCP solution side-step
> these problems? Any good ones out there?
> ____________________________________________________________ __________
> Apache Interface to OpenSSL (mod_ssl) www.modssl.org
> User Support Mailing List modssl-users@modssl.org
> Automated List Manager majordomo@modssl.org

--
Phil Ehrens | Fun stuff:
The LIGO Laboratory, MS 18-34 | http://www.ralphmag.org
California Institute of Technology | http://www.yellow5.com
1200 East California Blvd. | http://www.tokyotosho.com
Pasadena, CA 91125 USA | My gpg public key:
Phone:(626)395-8518 Fax:(626)793-9744 | http://www.imbe.net/peligo.asc
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

RE: CRL Checking Uses Excessive Memory

Hi Rob,

I also work for the DoD and am using the same CRLs as you (downloaded and
converted on a daily basis). We're running a Linux webserver with a single
1.8Ghz Celeron, 512MB of RAM, and 1GB of swap.

I haven't noticed any memory issues when checking CRLs.

My Apache server starts multiple child servers. It looks like the child
servers hit around 60MB of memory usage (max) when processing CRL checks;
500KB to 1MB seems to be the average child server's memory usage when idle.

top says my current load average is about 0.03, 0.01, 0.00. When checking
CRLs, top says my load average zooms up to around 0.20, 0.05, 0.01.

Of course, my userbase is very small and we aren't doing a ton of CRL
checks.

OCSP should resolve your issue with plowing through the CRLs, however, I
have yet to find a viable OCSP solution. There was a patch for mod_ssl, but
I haven't heard anything about it since it was last released in 2004. Maybe
someone else on this list knows?

Rob, why don't you email me offline. I'm in the DISA GAL, if you can get to
that.

Dwight...

-----Original Message-----
From: owner-modssl-users@modssl.org
[mailto:owner-modssl-users@modssl.org]On Behalf Of Walls Rob W Contr 75
CS/SCBS
Sent: Friday, April 21, 2006 10:47 AM
To: 'modssl-users@modssl.org'
Subject: CRL Checking Uses Excessive Memory


I work for the DoD. We have about a dozen CA's with their own CRL files.
Some of these are over 20M in size. When CRL checking is enabled in Apache
(for Linux or Windows), memory use is excessive and httpd processes are
killed by the OS (Linux) due to out of memory conditions and all the memory
swapping activity sends the proc utilization way up there and makes the
server unresponsive. On Windows the CPU use just pegs at 100% (I have no
idea what else is going on in there).
CRL's are downloaded every day and openssl is used to make hash'd file names
(ssl.conf is using SSLCARevocationPath). I don't currently restart apache
after retrieving the new CRL files.
The Linux machine runs redhat with dual 3ghz xeons and 2Gb ram. SSL works
great, but as soon as CRLs are checked, apache starts to go south! I have a
2Gb swap partition and have added another 2Gb swap file to at least keep
things running, but it becomes so slow it might as well crash.
Each httpd process goes from using about 14Mb of memory when not CRL
checking to 250Mb when CRL checking is enabled!
BTW: anywhere from 10 to 20 concurrent httpd processes are normal for that
machine.

Any ideas on how to use large CRL's in Apache?

Do I just need more memory?

If Apache can't use many large CRL files, would an OSCP solution side-step
these problems? Any good ones out there?
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

RE: CRL Checking Uses Excessive Memory

I am working on a DoD project, and we are experiencing high CPU load on HP-UX
servers with multiple CPUs in this scenario. We are thinking it is because
the CRL size for some CAs is huge - ad-hoc tests done with certs associated
with small CRLs do not produce CPU spikes, but large CRLs do. We are running
an older version of Apache and the mod_ssl package without OCSP support, but
have just installed an updated Apache with mod_ssl and OCSP support. Anyone
using this, and if so, have any luck with it? Thanks in advance!

Paul


Victor, Dwight P CTR DISA PAC wrote:
>
> Hi Rob,
>
> I also work for the DoD and am using the same CRLs as you (downloaded and
> converted on a daily basis). We're running a Linux webserver with a
> single
> 1.8Ghz Celeron, 512MB of RAM, and 1GB of swap.
>
> I haven't noticed any memory issues when checking CRLs.
>
> My Apache server starts multiple child servers. It looks like the child
> servers hit around 60MB of memory usage (max) when processing CRL checks;
> 500KB to 1MB seems to be the average child server's memory usage when
> idle.
>
> top says my current load average is about 0.03, 0.01, 0.00. When checking
> CRLs, top says my load average zooms up to around 0.20, 0.05, 0.01.
>
> Of course, my userbase is very small and we aren't doing a ton of CRL
> checks.
>
> OCSP should resolve your issue with plowing through the CRLs, however, I
> have yet to find a viable OCSP solution. There was a patch for mod_ssl,
> but
> I haven't heard anything about it since it was last released in 2004.
> Maybe
> someone else on this list knows?
>
> Rob, why don't you email me offline. I'm in the DISA GAL, if you can get
> to
> that.
>
> Dwight...
>
> -----Original Message-----
> From: owner-modssl-users@modssl.org
> [mailto:owner-modssl-users@modssl.org]On Behalf Of Walls Rob W Contr 75
> CS/SCBS
> Sent: Friday, April 21, 2006 10:47 AM
> To: 'modssl-users@modssl.org'
> Subject: CRL Checking Uses Excessive Memory
>
>
> I work for the DoD. We have about a dozen CA's with their own CRL files.
> Some of these are over 20M in size. When CRL checking is enabled in Apache
> (for Linux or Windows), memory use is excessive and httpd processes are
> killed by the OS (Linux) due to out of memory conditions and all the
> memory
> swapping activity sends the proc utilization way up there and makes the
> server unresponsive. On Windows the CPU use just pegs at 100% (I have no
> idea what else is going on in there).
> CRL's are downloaded every day and openssl is used to make hash'd file
> names
> (ssl.conf is using SSLCARevocationPath). I don't currently restart apache
> after retrieving the new CRL files.
> The Linux machine runs redhat with dual 3ghz xeons and 2Gb ram. SSL works
> great, but as soon as CRLs are checked, apache starts to go south! I have
> a
> 2Gb swap partition and have added another 2Gb swap file to at least keep
> things running, but it becomes so slow it might as well crash.
> Each httpd process goes from using about 14Mb of memory when not CRL
> checking to 250Mb when CRL checking is enabled!
> BTW: anywhere from 10 to 20 concurrent httpd processes are normal for that
> machine.
>
> Any ideas on how to use large CRL's in Apache?
>
> Do I just need more memory?
>
> If Apache can't use many large CRL files, would an OSCP solution side-step
> these problems? Any good ones out there?
> ____________________________________________________________ __________
> Apache Interface to OpenSSL (mod_ssl) www.modssl.org
> User Support Mailing List modssl-users@modssl.org
> Automated List Manager majordomo@modssl.org
> ____________________________________________________________ __________
> Apache Interface to OpenSSL (mod_ssl) www.modssl.org
> User Support Mailing List modssl-users@modssl.org
> Automated List Manager majordomo@modssl.org
>
>

--
View this message in context: http://www.nabble.com/CRL-Checking-Uses-Excessive-Memory-tf1 488925.html#a6764331
Sent from the mod_ssl - Users mailing list archive at Nabble.com.

____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org