Apache is slow at accepting connections

Hello,
we have a rather busy Apache web server (~200/300 contemporary
connections). There are time when Apache is really slow at letting
client connects to it. For example, with curl, I see:

02:05:17.885074 == Info: About to connect() to IP_ADDRESS port 80 (#0)
02:05:17.885280 == Info: Trying IP_ADDRESS... 02:05:20.898748 == Info:
connected
02:05:20.898785 == Info: Connected to IP_ADDRESS (IP_ADDRESS) port 80 (#0)
....
02:05:20.917068 == Info: Closing connection #0

and

02:06:53.098230 == Info: About to connect() to IP_ADDRESS port 80 (#0)
02:06:53.099272 == Info: Trying IP_ADDRESS... 02:07:02.111596 == Info:
connected
02:07:02.111636 == Info: Connected to IP_ADDRESS (IP_ADDRESS) port 80 (#0)
02:07:02.111731 => Send header, 222 bytes (0xde)
....
02:07:02.422093 == Info: Closing connection #0

as you can see, we see a 3 seconds (first example) and a 9 seconds
(second example) delay between apache server contact and when the
connection is actually accepted. The delay is always either 3 or 9
seconds, that's quite weird and it seems to indicate a sort of
retry+backoff (3 secs, 3x2 secs (9 secs total) and so on) implemented in
some Apache layers.

Anyhow, once the client is able to connect to Apache, the request is
served very fast. Just to be sure it's nothing in the VirtualHosts we've
setup, we tried also with /server-status but even that page is slow at
connect.

In addition, sometimes we also see:

02:10:40.651926 == Info: About to connect() to IP_ADDRESS port 80 (#0)
02:10:40.652543 == Info: Trying IP_ADDRESS... 02:11:00.060879 == Info:
Connection refused
02:11:00.060924 == Info: couldn't connect to host
02:11:00.060942 == Info: Closing connection #0

as if the client was not able to be served by any Apache server.

The MPM config we use is:

StartServers 200
MinSpareServers 150
MaxSpareServers 300
ServerLimit 2000
MaxClients 2000
MaxRequestsPerChild 100

KeepAlive off

We tried everything that came to our minds to avoid that delay:

- tune the parameters above (StartServer, Min/MaxSpareServer etc)
- only Listen on a single port (inferred from
http://httpd.apache.org/docs/2.2/misc/perf-tuning.html)
- change every possible AcceptMutex values: flock, fcntl, pthread, and
sysvsem (the default)

but none of them helped.

Does anyone know how to solve this problem? If there are other info
needed, just let me now and I'll provide them.

Thanks in advance,
Sandro

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

RE: Apache is slow at accepting connections

> -----Message d'origine-----
> De=A0: Sandro Tosi [mailto:sandro.tosi@register.it]
> Envoyé : mardi 2 f=E9vrier 2010 15:00
> À : users@httpd.apache.org
> Objet=A0: [users@httpd] Apache is slow at accepting connections
>=20
> Hello,
> we have a rather busy Apache web server (~200/300 contemporary
> connections). There are time when Apache is really slow at letting
> client connects to it. For example, with curl, I see:
>=20
> 02:05:17.885074 == Info: About to connect() to IP_ADDRESS port 80 (#0=
)
> 02:05:17.885280 == Info: Trying IP_ADDRESS... 02:05:20.898748 =3D=
=3D Info:
> connected
> 02:05:20.898785 == Info: Connected to IP_ADDRESS (IP_ADDRESS) port 80=
(#0)
> ...
> 02:05:20.917068 == Info: Closing connection #0
>=20
> and
>=20
> 02:06:53.098230 == Info: About to connect() to IP_ADDRESS port 80 (#0=
)
> 02:06:53.099272 == Info: Trying IP_ADDRESS... 02:07:02.111596 =3D=
=3D Info:
> connected
> 02:07:02.111636 == Info: Connected to IP_ADDRESS (IP_ADDRESS) port 80=
(#0)
> 02:07:02.111731 =3D> Send header, 222 bytes (0xde)
> ...
> 02:07:02.422093 == Info: Closing connection #0
>=20
> as you can see, we see a 3 seconds (first example) and a 9 seconds
> (second example) delay between apache server contact and when the
> connection is actually accepted. The delay is always either 3 or 9
> seconds, that's quite weird and it seems to indicate a sort of
> retry+backoff (3 secs, 3x2 secs (9 secs total) and so on) implemented in
> some Apache layers.
>=20
> Anyhow, once the client is able to connect to Apache, the request is
> served very fast. Just to be sure it's nothing in the VirtualHosts we've
> setup, we tried also with /server-status but even that page is slow at
> connect.
>=20
> In addition, sometimes we also see:
>=20
> 02:10:40.651926 == Info: About to connect() to IP_ADDRESS port 80 (#0=
)
> 02:10:40.652543 == Info: Trying IP_ADDRESS... 02:11:00.060879 =3D=
=3D Info:
> Connection refused
> 02:11:00.060924 == Info: couldn't connect to host
> 02:11:00.060942 == Info: Closing connection #0
>=20
> as if the client was not able to be served by any Apache server.
>=20
> The MPM config we use is:
>=20
> StartServers 200
> MinSpareServers 150
> MaxSpareServers 300
> ServerLimit 2000
> MaxClients 2000
> MaxRequestsPerChild 100
>=20
> KeepAlive off
>=20
> We tried everything that came to our minds to avoid that delay:
>=20
> - tune the parameters above (StartServer, Min/MaxSpareServer etc)
> - only Listen on a single port (inferred from
> http://httpd.apache.org/docs/2.2/misc/perf-tuning.html)
> - change every possible AcceptMutex values: flock, fcntl, pthread, and
> sysvsem (the default)
>=20
> but none of them helped.
>=20
> Does anyone know how to solve this problem? If there are other info
> needed, just let me now and I'll provide them.
>=20
> Thanks in advance,
> Sandro
>=20

Hi,

Couldn't this latency come from dns lookups ?
Are you using some directives such as 'UseCanonicalName DNS', 'HostnameLook=
ups On|double' or even hostnames in some 'VirtualHost' entries ?

Emmanuel

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache is slow at accepting connections

On Tue, Feb 2, 2010 at 8:59 AM, Sandro Tosi wrote=
:
> Hello,
> we have a rather busy Apache web server (~200/300 contemporary connection=
s).
> There are time when Apache is really slow at letting client connects to i=
t.
> For example, with curl, I see:
>
> 02:05:17.885074 == Info: About to connect() to IP_ADDRESS port 80 (#0=
)
> 02:05:17.885280 == Info: =A0 Trying IP_ADDRESS... 02:05:20.898748 =3D=
=3D Info:
> connected
> 02:05:20.898785 == Info: Connected to IP_ADDRESS (IP_ADDRESS) port 80=
(#0)
> ...
> 02:05:20.917068 == Info: Closing connection #0
>
> and
>
> 02:06:53.098230 == Info: About to connect() to IP_ADDRESS port 80 (#0=
)
> 02:06:53.099272 == Info: =A0 Trying IP_ADDRESS... 02:07:02.111596 =3D=
=3D Info:
> connected
> 02:07:02.111636 == Info: Connected to IP_ADDRESS (IP_ADDRESS) port 80=
(#0)
> 02:07:02.111731 =3D> Send header, 222 bytes (0xde)
> ...
> 02:07:02.422093 == Info: Closing connection #0
>
> as you can see, we see a 3 seconds (first example) and a 9 seconds (secon=
d
> example) delay between apache server contact and when the connection is
> actually accepted. The delay is always either 3 or 9 seconds, that's quit=
e
> weird and it seems to indicate a sort of retry+backoff (3 secs, 3x2 secs =
(9
> secs total) and so on) implemented in some Apache layers.

Getting connected and sending the first bytes on a connection is
handled by the TCP layer and below; Apache doesn't have to get
involved yet to allow that to happen.

Apache does give the TCP layer a hint about how to behave in this part
of the request cycle -- ListenBacklog. You could see if increasing
that setting in Apache convinces the TCP layer on the Apache side to
accept more connections even before Apache wakes up to process them.

(A TCP trace would presumably confirm that getting the connect
handshake completed in a timely manner is the issue. It might also
show something totally unanticipated.)

> StartServers 200
> MinSpareServers 150
> MaxSpareServers 300

Given your comment about 200-300 typical active connections, I guess
this is ok if your traffic isn't at all spiky.

> ServerLimit 2000
> MaxClients 2000

definitely not a problem here

> MaxRequestsPerChild 100

This will cause httpd processes to exit relatively quickly and might
occasionally increase the amount of time before Apache is able to
accept a new connection

It is very hard to say for sure that this can; I guess increasing
MaxSpareServers could mitigate the potential for temporary windows
where a lot more Apache processes have to be created at once due to
exiting processes. (I wonder how MinSpare../MaxSpare../MaxRequests..
interact with your actual load.)

Anyway, MaxRequestsPerChild rarely needs to be set so low; that would
be needed only with barely working application code running inside
Apache. Unless you have a real reason to set it so low, either
disable it (0) or set it very high (at least some tens of thousands
for the prefork MPM).

>
> KeepAlive off

This results in more connections required for real browser traffic,
exacerbating any connection problem.

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache is slow at accepting connections

Hello Emmanuel,
thanks for your reply.

Emmanuel Bailleul wrote:
> Hi,
>
> Couldn't this latency come from dns lookups ?
>
If that's the case, I'd expect to see the delay at each request; I do my
tests on a single machine and I see results with no delay, 3 secs delay,
9 secs delay and conn-timeout. If we have dns-reverse lookup delay I'd
expect it to be a constant time during tests, and also we have dns cache
on the servers, so it should (at least) happen just once and then the
entry be cached locally and not asked to dns server.
> Are you using some directives such as 'UseCanonicalName DNS', 'HostnameLookups On|double' or even hostnames in some 'VirtualHost' entries ?
>
None of them are set or set to 'Off'.

Regards,
Sandro

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache is slow at accepting connections

Hello Jeff,
thanks a lot for your insightful reply!

Jeff Trawick wrote:
> On Tue, Feb 2, 2010 at 8:59 AM, Sandro Tosi wrote:
>
>> Hello,
>> we have a rather busy Apache web server (~200/300 contemporary connections).
>> There are time when Apache is really slow at letting client connects to it.
>> For example, with curl, I see:
>>
>> 02:05:17.885074 == Info: About to connect() to IP_ADDRESS port 80 (#0)
>> 02:05:17.885280 == Info: Trying IP_ADDRESS... 02:05:20.898748 == Info:
>> connected
>> 02:05:20.898785 == Info: Connected to IP_ADDRESS (IP_ADDRESS) port 80 (#0)
>> ...
>> 02:05:20.917068 == Info: Closing connection #0
>>
>> and
>>
>> 02:06:53.098230 == Info: About to connect() to IP_ADDRESS port 80 (#0)
>> 02:06:53.099272 == Info: Trying IP_ADDRESS... 02:07:02.111596 == Info:
>> connected
>> 02:07:02.111636 == Info: Connected to IP_ADDRESS (IP_ADDRESS) port 80 (#0)
>> 02:07:02.111731 => Send header, 222 bytes (0xde)
>> ...
>> 02:07:02.422093 == Info: Closing connection #0
>>
>> as you can see, we see a 3 seconds (first example) and a 9 seconds (second
>> example) delay between apache server contact and when the connection is
>> actually accepted. The delay is always either 3 or 9 seconds, that's quite
>> weird and it seems to indicate a sort of retry+backoff (3 secs, 3x2 secs (9
>> secs total) and so on) implemented in some Apache layers.
>>
>
> Getting connected and sending the first bytes on a connection is
> handled by the TCP layer and below; Apache doesn't have to get
> involved yet to allow that to happen.
>
Ok, now I see.
> Apache does give the TCP layer a hint about how to behave in this part
> of the request cycle -- ListenBacklog. You could see if increasing
> that setting in Apache convinces the TCP layer on the Apache side to
> accept more connections even before Apache wakes up to process them.
>
We'll give it a look for sure. We looked it up on the doc and the
ListenBacklog entry is quite "short": we're on Linux, do you have any
suggestions or links for more info on it?
> (A TCP trace would presumably confirm that getting the connect
> handshake completed in a timely manner is the issue. It might also
> show something totally unanticipated.)
>
any hint how to obtain that TCP trace? tcpdump on the client (easy) or
on the server (almost impossible due to the high traffic)?
>
>> StartServers 200
>> MinSpareServers 150
>> MaxSpareServers 300
>>
>
> Given your comment about 200-300 typical active connections, I guess
> this is ok if your traffic isn't at all spiky.
>
we also tried to increase those values a bit

StartServers 300
MinSpareServers 200
MaxSpareServers 400

but it seems slowing down (probably due to the high number of process to
handle); we'll probably revert to the above settings.
>
>> ServerLimit 2000
>> MaxClients 2000
>>
>
> definitely not a problem here
>
yeah, we barely reach the 1.5 * StartServer number of servers,
definitely under those limits.
>
>> MaxRequestsPerChild 100
>>
>
> This will cause httpd processes to exit relatively quickly and might
> occasionally increase the amount of time before Apache is able to
> accept a new connection
>
> It is very hard to say for sure that this can; I guess increasing
> MaxSpareServers could mitigate the potential for temporary windows
> where a lot more Apache processes have to be created at once due to
> exiting processes. (I wonder how MinSpare../MaxSpare../MaxRequests..
> interact with your actual load.)
>
> Anyway, MaxRequestsPerChild rarely needs to be set so low; that would
> be needed only with barely working application code running inside
> Apache. Unless you have a real reason to set it so low, either
> disable it (0) or set it very high (at least some tens of thousands
> for the prefork MPM).
>
Ah. We were under the false impression that setting it to a low value it
helps apache run on "fresh" processes, hence better, but we
underestimate the kill&spawn time.

We are running Apache as a reverse proxy for a JBoss instance, so we can
expect almost no memory leak, so we'll try either with 0 or with a very
high value.
>> KeepAlive off
>>
>
> This results in more connections required for real browser traffic,
> exacerbating any connection problem.
we don't have a "real browser" traffic: we receive just a request (for a
single resource) for each client, so it's a sort-of "one shot" traffic
not a continuous flow of requests from the same client. Do you think
it's ok to have the KA Off for such a kind of traffic?

Thanks & Regards,
Sandro

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache is slow at accepting connections

--------------------------------------------------
From: "Sandro Tosi"
Sent: 02 February, 2010 15:58
To:
Subject: Re: [users@httpd] Apache is slow at accepting connections


>
> We are running Apache as a reverse proxy for a JBoss instance, so we can
> expect almost no memory leak, so we'll try either with 0 or with a very high
> value.

Sandro,

We use various applications at work that use JBoss, and I see the samethings you
see. I would suspect that it is an issue with Jboss. At times we have to call
in support to restart the hung jboss services to enable faster connection times.

Do you exhibit this same delay on non-jboss served pages?

Thanks,
Daniel


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache is slow at accepting connections

Hi Daniel,
thanks for your reply.

Daniel Reinhardt wrote:
>> We are running Apache as a reverse proxy for a JBoss instance, so we can
>> expect almost no memory leak, so we'll try either with 0 or with a very high
>> value.
>>
>
> Sandro,
>
> We use various applications at work that use JBoss, and I see the samethings you
> see. I would suspect that it is an issue with Jboss. At times we have to call
> in support to restart the hung jboss services to enable faster connection times.
>
>
Our problem is not JBoss, luckily :) Calling directly the java app it
replies fast.
> Do you exhibit this same delay on non-jboss served pages?
>
Yes: we conduct the tests on Apache only calling the server-status page
(ok, it's not "that" static, but it still just queries Apache internal
information to return the status page) and we see the same behavior.

Regards,
Sandro

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: Apache is slow at accepting connections

Hello,
we managed to fix this problem by setting:

net.ipv4.tcp_syncookies=1
net.ipv4.tcp_tw_reuse=1

kernel params. tcp_syscookies is enough but since we had several
connections in TIME_WAIT, we read tcp_tw_reuse helps those be reused
more efficiently.

HTH,
Sandro

Sandro Tosi wrote:
> Hello,
> we have a rather busy Apache web server (~200/300 contemporary
> connections). There are time when Apache is really slow at letting
> client connects to it. For example, with curl, I see:
>
> 02:05:17.885074 == Info: About to connect() to IP_ADDRESS port 80 (#0)
> 02:05:17.885280 == Info: Trying IP_ADDRESS... 02:05:20.898748 == Info:
> connected
> 02:05:20.898785 == Info: Connected to IP_ADDRESS (IP_ADDRESS) port 80 (#0)
> ...
> 02:05:20.917068 == Info: Closing connection #0
>
> and
>
> 02:06:53.098230 == Info: About to connect() to IP_ADDRESS port 80 (#0)
> 02:06:53.099272 == Info: Trying IP_ADDRESS... 02:07:02.111596 == Info:
> connected
> 02:07:02.111636 == Info: Connected to IP_ADDRESS (IP_ADDRESS) port 80 (#0)
> 02:07:02.111731 => Send header, 222 bytes (0xde)
> ...
> 02:07:02.422093 == Info: Closing connection #0
>
> as you can see, we see a 3 seconds (first example) and a 9 seconds
> (second example) delay between apache server contact and when the
> connection is actually accepted. The delay is always either 3 or 9
> seconds, that's quite weird and it seems to indicate a sort of
> retry+backoff (3 secs, 3x2 secs (9 secs total) and so on) implemented in
> some Apache layers.
>
> Anyhow, once the client is able to connect to Apache, the request is
> served very fast. Just to be sure it's nothing in the VirtualHosts we've
> setup, we tried also with /server-status but even that page is slow at
> connect.
>
> In addition, sometimes we also see:
>
> 02:10:40.651926 == Info: About to connect() to IP_ADDRESS port 80 (#0)
> 02:10:40.652543 == Info: Trying IP_ADDRESS... 02:11:00.060879 == Info:
> Connection refused
> 02:11:00.060924 == Info: couldn't connect to host
> 02:11:00.060942 == Info: Closing connection #0
>
> as if the client was not able to be served by any Apache server.
>
> The MPM config we use is:
>
> StartServers 200
> MinSpareServers 150
> MaxSpareServers 300
> ServerLimit 2000
> MaxClients 2000
> MaxRequestsPerChild 100
>
> KeepAlive off
>
> We tried everything that came to our minds to avoid that delay:
>
> - tune the parameters above (StartServer, Min/MaxSpareServer etc)
> - only Listen on a single port (inferred from
> http://httpd.apache.org/docs/2.2/misc/perf-tuning.html)
> - change every possible AcceptMutex values: flock, fcntl, pthread, and
> sysvsem (the default)
>
> but none of them helped.
>
> Does anyone know how to solve this problem? If there are other info
> needed, just let me now and I'll provide them.
>
> Thanks in advance,
> Sandro
>
> ------------------------------------------------------------ ---------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org