slapper(?) causing DoS/mutex file disappearing

Starting last Thursday, we started to see one of our webservers
become unresponsive for about 10 minutes...it seemed to be correlated
with what appeared to be a slapper/OpenSSL worm attack. We are
not vulnerable to the worm but the attack seemed to use up some
resources (not CPU) that prevented apache from answering more requests.
Note that it corrects itself after 10 minutes or so without manual
intervention.

Here's the ouput of our Server: header.
Server: Apache/1.3.26 (Unix) mod_ssl/2.8.9 OpenSSL/0.9.6g mod_jk

The error in the logs is:
[Thu Sep 26 20:55:18 2002] [error] OpenSSL: error:1406B458:SSL routines:GET_CLIENT_MASTER_KEY:key arg too long

There also are a lot of errors like this that start at the same time:
[Thu Sep 26 20:49:36 2002] [error] mod_ssl: Child could not open SSLMutex lockfile /usr/local/apache/logs/ssl_mutex.22003 (System error follows)

And sure enough the mutex file on that server is gone. It comes
back on restart...but what the heck is going on here? Anyone having
similar issues?

This is driving me crazy as this is on our production servers and
I'm not going to get a wink of sleep tonight unless I figure out
how to stop it....

____________________________________________________________ _______
P a u l
g@gunman.org
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

Re: slapper(?) causing DoS/mutex file disappearing

On Sunday 29 September 2002 21:54, P a u l Guth wrote:

we are experiencing the same here esp. on machines with lots of normal
apache-clients + lots of ips.

I guess that Apache does detect the problem and writes

client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /

to the logs but does not terminate the child-process. So at one time -when
you have a lot of ips and they are all scanned - you reach the MaxClients
limit.

I'm not sure why apache behaves this way.

Andreas

> Starting last Thursday, we started to see one of our webservers
> become unresponsive for about 10 minutes...it seemed to be correlated
> with what appeared to be a slapper/OpenSSL worm attack. We are
> not vulnerable to the worm but the attack seemed to use up some
> resources (not CPU) that prevented apache from answering more requests.
> Note that it corrects itself after 10 minutes or so without manual
> intervention.
>
> Here's the ouput of our Server: header.
> Server: Apache/1.3.26 (Unix) mod_ssl/2.8.9 OpenSSL/0.9.6g mod_jk
>
> The error in the logs is:
> [Thu Sep 26 20:55:18 2002] [error] OpenSSL: error:1406B458:SSL
> routines:GET_CLIENT_MASTER_KEY:key arg too long
>
> There also are a lot of errors like this that start at the same time:
> [Thu Sep 26 20:49:36 2002] [error] mod_ssl: Child could not open SSLMutex
> lockfile /usr/local/apache/logs/ssl_mutex.22003 (System error follows)
>
> And sure enough the mutex file on that server is gone. It comes
> back on restart...but what the heck is going on here? Anyone having
> similar issues?
>
> This is driving me crazy as this is on our production servers and
> I'm not going to get a wink of sleep tonight unless I figure out
> how to stop it....
>
> ____________________________________________________________ _______
> P a u l
> g@gunman.org
> ____________________________________________________________ __________
> Apache Interface to OpenSSL (mod_ssl) www.modssl.org
> User Support Mailing List modssl-users@modssl.org
> Automated List Manager majordomo@modssl.org

--
e-admin internet gmbh
Andreas Gietl tel +49 941 3810884
Ludwig-Thoma-Strasse 35 fax +49 941 3810891
93051 Regensburg mobil +49 171 6070008

PGP/GPG-Key unter http://www.e-admin.de/gpg.html




____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

Re: slapper(?) causing DoS/mutex file disappearing

On Sun, Sep 29, 2002 at 12:54:02PM -0700, P a u l Guth wrote:
> Starting last Thursday, we started to see one of our webservers
> become unresponsive for about 10 minutes...it seemed to be correlated
> with what appeared to be a slapper/OpenSSL worm attack. We are
> not vulnerable to the worm but the attack seemed to use up some
> resources (not CPU) that prevented apache from answering more requests.
> Note that it corrects itself after 10 minutes or so without manual
> intervention.
>
> Here's the ouput of our Server: header.
> Server: Apache/1.3.26 (Unix) mod_ssl/2.8.9 OpenSSL/0.9.6g mod_jk
>
> The error in the logs is:
> [Thu Sep 26 20:55:18 2002] [error] OpenSSL: error:1406B458:SSL routines:GET_CLIENT_MASTER_KEY:key arg too long
>
> There also are a lot of errors like this that start at the same time:
> [Thu Sep 26 20:49:36 2002] [error] mod_ssl: Child could not open SSLMutex lockfile /usr/local/apache/logs/ssl_mutex.22003 (System error follows)
>
> And sure enough the mutex file on that server is gone. It comes
> back on restart...but what the heck is going on here? Anyone having
> similar issues?
>
> This is driving me crazy as this is on our production servers and
> I'm not going to get a wink of sleep tonight unless I figure out
> how to stop it....
>

slapper if is not successful chokes your Web Server into overload.

solutions:

1) Limit you Max s! I limited my MaxKeepAliveRequests from 100 to 20.
2) If on Cisco, use rate-Limiting. Check http://www.cisco.com/warp/public/707/newsflash.html .

Painful meds but this is goingto be nasty!!
> ____________________________________________________________ _______
> P a u l
> g@gunman.org
> ____________________________________________________________ __________
> Apache Interface to OpenSSL (mod_ssl) www.modssl.org
> User Support Mailing List modssl-users@modssl.org
> Automated List Manager majordomo@modssl.org

--
Member - Liberal International On 11 Sept 2001 the WORLD was violated.
This is doctor@nl2k.ab.ca Ici doctor@nl2k.ab.ca
Society MUST be saved! Extremists must dissolve.
Beware of defining as intelligent only those who share your opinions
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

Re: slapper(?) causing DoS/mutex file disappearing

FYI, I believe I have tracked this problem down to the Mutex file
being deleted during routine maintenance. Since recreating the Mutex
file (via restart) I haven't seen the problem recur, and there have
been a few hits from the worm.

Sorry for the alarm. Nothing to see here.

On Sun, Sep 29, 2002 at 12:54:02PM -0700, P a u l Guth wrote:
> Starting last Thursday, we started to see one of our webservers
> become unresponsive for about 10 minutes...it seemed to be correlated
> with what appeared to be a slapper/OpenSSL worm attack. We are
> not vulnerable to the worm but the attack seemed to use up some
> resources (not CPU) that prevented apache from answering more requests.
> Note that it corrects itself after 10 minutes or so without manual
> intervention.
>
> Here's the ouput of our Server: header.
> Server: Apache/1.3.26 (Unix) mod_ssl/2.8.9 OpenSSL/0.9.6g mod_jk
>
> The error in the logs is:
> [Thu Sep 26 20:55:18 2002] [error] OpenSSL: error:1406B458:SSL routines:GET_CLIENT_MASTER_KEY:key arg too long
>
> There also are a lot of errors like this that start at the same time:
> [Thu Sep 26 20:49:36 2002] [error] mod_ssl: Child could not open SSLMutex lockfile /usr/local/apache/logs/ssl_mutex.22003 (System error follows)
>
> And sure enough the mutex file on that server is gone. It comes
> back on restart...but what the heck is going on here? Anyone having
> similar issues?
>
> This is driving me crazy as this is on our production servers and
> I'm not going to get a wink of sleep tonight unless I figure out
> how to stop it....
>
> ____________________________________________________________ _______
> P a u l
> g@gunman.org
>

--
____________________________________________________________ _______
P a u l
g@gunman.org
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org