Apache Close-Wait ports issue

------_=_NextPart_001_01CA587C.769EA4CD
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi All

=20

Got an interesting problem which has occurred several times now..

=20

Painting a brief picture ;

=20

Several applications running on Tomcat 6.0.19 (64-bit) - several cluster
groups

Backend SQL Server 2005 clustered DBs (several)

Front end Apache Web servers (2.2.9 with mod_jk 1.2.19), load balanced
via secure network appliance

=20

Basically, we have an issue with one of the DB clusters which stops
servicing requests from the app servers

The app servers can continue to connect to remaining DB clusters OK.

=20

The problem is that all the web servers stop responding to external
requests. Looking at the ports in use on each server, there are a
number of established sessions between Apache and Tomcat, which seems
OK. All of the web servers also have exactly 30 close-wait sessions
with the secure network appliance. We think this is what is stopping
any requests from being services by the web servers. However, even
trying to browse locally on each server to the root just times out.

=20

We do not enforce keepalive or timeouts for TCP/HTTP, and we use
practically all the defaults for these directives. All web servers are
configures identically.

=20

I can't find any reference to a directive that may limit connections to
30.

=20

Restarting the web servers allows users to get to applications using the
good databases. We have already upgraded one of the load balanced web
servers to Apache 2.2.14 with mod_jk 1.2.28, and we intend to do the
rest of the next few days. Going through the changelogs there are one
or two items that might have a bearing on the issue but nothing obvious.

=20

Does anybody have any advice on this?

=20

Cheers,

Darren.

=20



Connaught plc is a FTSE 250 company. We are the UK's leading provider of in=
tegrated services operating in the compliance, social housing and public se=
ctor markets.

Please visit our website to see a full list of Connaught's Registered Compa=
nies www.connaught.plc.uk/group/aboutconnaught/registeredcompanie s

Disclaimer:
The information transmitted is intended only for the person or entity to wh=
ich it is addressed and may contain confidential and/or privileged material=
.. Any review, retransmission, dissemination or other use of, or taking of a=
ny action in reliance upon, this information by persons or entities other t=
han the intended recipient is prohibited. If you received this in error, pl=
ease contact the sender and delete this message. Connaught plc, Head Office=
01392 444546

------_=_NextPart_001_01CA587C.769EA4CD
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">

Re: Apache Close-Wait ports issue

Welcome ;-)

Darren Kukulka wrote:
>
> Painting a brief picture ;
> Several applications running on Tomcat 6.0.19 (64-bit) - several cluster
> groups
> Backend SQL Server 2005 clustered DBs (several)
> Front end Apache Web servers (2.2.9 with mod_jk 1.2.19), load balanced
> via secure network appliance
>
....
> The problem is that all the web servers stop responding to external
> requests. Looking at the ports in use on each server...
.. All of the web servers also have exactly 30 close-wait sessions
> with the secure network appliance.


If I get this right, you have

browsers <--> network appliance <--> Apache httpd servers <--> other
stuff
and your CLOSE_WAIT sockets are on the Apache httpd servers, for
connections with the network appliance, right ?

Did you check (with "netstat -pan" e.g.) if that is really the case ?

If that is correct, it would mean that Apache has closed these sockets,
but is waiting for the network appliance to close it's side of these
connections, which never happens. Any idea why that might be ?

Another question : when that happens, how many Apache children/threads
are running ?
Maybe you just have 30 Apache children/threads that are "stuck" because
they are waiting for the network appliance to close these sockets, and
it never happens.
As long as they would be waiting for that, they would never be recycled
by Apache to handle new requests, which may explain what you're seeing ?



------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

RE: Apache Close-Wait ports issue

Thanks for the reply Andre...responses in line

> -----Original Message-----
> From: Andr=E9 Warnier [mailto:aw@ice-sa.com]
> Sent: 29 October 2009 10:47
> To: users@httpd.apache.org
> Subject: Re: [users@httpd] Apache Close-Wait ports issue
>=20
> Welcome ;-)
>=20
> Darren Kukulka wrote:
> >
> > Painting a brief picture ;
> > Several applications running on Tomcat 6.0.19 (64-bit) - several
> cluster
> > groups
> > Backend SQL Server 2005 clustered DBs (several)
> > Front end Apache Web servers (2.2.9 with mod_jk 1.2.19), load
> balanced
> > via secure network appliance
> >
> ...
> > The problem is that all the web servers stop responding to external
> > requests. Looking at the ports in use on each server...
> . All of the web servers also have exactly 30 close-wait sessions
> > with the secure network appliance.
>=20
>=20
> If I get this right, you have
>=20
> browsers <--> network appliance <--> Apache httpd servers <--> other
> stuff
> and your CLOSE_WAIT sockets are on the Apache httpd servers, for
> connections with the network appliance, right ?
>=20


Correct - the connections from the Web Servers to the Tomcat app servers do=
wn the line remain established.

> Did you check (with "netstat -pan" e.g.) if that is really the case ?
>=20
> If that is correct, it would mean that Apache has closed these sockets,
> but is waiting for the network appliance to close it's side of these
> connections, which never happens. Any idea why that might be ?
>=20
> Another question : when that happens, how many Apache children/threads
> are running ?
> Maybe you just have 30 Apache children/threads that are "stuck" because
> they are waiting for the network appliance to close these sockets, and
> it never happens.
> As long as they would be waiting for that, they would never be recycled
> by Apache to handle new requests, which may explain what you're seeing
> ?
>

So as we're not specifying any directives in the httpd.conf, with respect t=
o connection control, the default of KeepAlive=3Don and KeepAliveTimout=3D1=
5 applies? Would this not clear out the close-wait connections?

We've queried the network appliance vendor but they are unaware of any reas=
ons the appliances would not close the sockets.
=20
>=20
>=20
> ------------------------------------------------------------ ---------
> The official User-To-User support forum of the Apache HTTP Server
> Project.
> See for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org



Connaught plc is a FTSE 250 company. We are the UK's leading provider of in=
tegrated services operating in the compliance, social housing and public se=
ctor markets.

Please visit our website to see a full list of Connaught's Registered Compa=
nies www.connaught.plc.uk/group/aboutconnaught/registeredcompanie s

Disclaimer:
The information transmitted is intended only for the person or entity to wh=
ich it is addressed and may contain confidential and/or privileged material=
.. Any review, retransmission, dissemination or other use of, or taking of a=
ny action in reliance upon, this information by persons or entities other t=
han the intended recipient is prohibited. If you received this in error, pl=
ease contact the sender and delete this message. Connaught plc, Head Office=
01392 444546

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache Close-Wait ports issue

Darren Kukulka wrote:
>
> So as we're not specifying any directives in the httpd.conf, with respect to connection control, the default of KeepAlive=on and KeepAliveTimout=15 applies? Would this not clear out the close-wait connections?
>
No.
KeepAliveTimeout 15
is *very* long nowadays. It means that, after a browser has sent its
last request (and is maybe never going to make another request), the
Apache server's child/thread which was handling that connection, will
wait for up to 15 seconds, doing nothing, before it decides that the
browser is never going to ask for something else, and then close its
side of the connection and, possibly, signal that it is now available to
handle another request.

Try setting to 3 for instance.



------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org