mod_proxy_ajp (2.2.13) + Tomcat 6.0.20 - APR does not understand this

Hi,
we have one Apache-2.2.13 running mod_proxy_ajp + mod_proxy_balancer,
connected to (3) Tomcat-6.0.20 instances under Fedora release 8. We
are experiencing some issues with high CPU load on the Tomcat side,
and Apache starts logging errors like these

[Fri Oct 30 14:47:43 2009] [error] (70007)The timeout specified has
expired: ajp_ilink_receive() can't receive header
[Fri Oct 30 14:47:43 2009] [error] ajp_read_header: ajp_ilink_receive failed
[Fri Oct 30 14:47:43 2009] [error] (120006)APR does not understand
this error code: proxy: read response failed from

On Tomcat side we are using the APR Native Connector, with this settings:

maxThreads="130" backlog="10" connectionTimeout="30000"
keepAliveTimeout="30000" enableLookups="false" />

Apache MPM conf:

ServerLimit 100

StartServers 30
MaxClients 2500
MinSpareThreads 50
MaxSpareThreads 100
ThreadsPerChild 25
MaxRequestsPerChild 0


mod_proxy_ajp conf:


BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app01 ttl=30 max=2
BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app02 ttl=30 max=2
BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app03 ttl=30 max=2


Apache specs:

Server version: Apache/2.2.13 (Unix)
Server built: Sep 29 2009 11:12:35
Server's Module Magic Number: 20051115:23
Server loaded: APR 1.3.8, APR-Util 1.3.9
Compiled using: APR 1.3.8, APR-Util 1.3.9
Architecture: 32-bit
Server MPM: Worker
threaded: yes (fixed thread count)
forked: yes (variable process count)


I've beeng testing different settings for mod_proxy_ajp, but I am not
sure what is the exact relation between the max option for mod_proxy
and ThreadsPerChild. Should I set max to 1? I understand I would have
as many connections to each Tomcat box as
max*(MaxClients/ThreadsPerChild), is this correct?

BTW, is the total number of connections that mod_proxy_ajp will
forward to Tomcat related with MaxThreads ? I don't fully understand
how APR connector works in this case.

Thanks in advance,
-Damian Traverso

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: mod_proxy_ajp (2.2.13) + Tomcat 6.0.20 - APR doesnot understand this error code: proxy: read re

Damian Traverso wrote:
> Hi,
> we have one Apache-2.2.13 running mod_proxy_ajp + mod_proxy_balancer,
> connected to (3) Tomcat-6.0.20 instances under Fedora release 8. We
> are experiencing some issues with high CPU load on the Tomcat side,
> and Apache starts logging errors like these
>
> [Fri Oct 30 14:47:43 2009] [error] (70007)The timeout specified has
> expired: ajp_ilink_receive() can't receive header
> [Fri Oct 30 14:47:43 2009] [error] ajp_read_header: ajp_ilink_receive failed
> [Fri Oct 30 14:47:43 2009] [error] (120006)APR does not understand
> this error code: proxy: read response failed from
>
> On Tomcat side we are using the APR Native Connector, with this settings:
>
> > maxThreads="130" backlog="10" connectionTimeout="30000"
> keepAliveTimeout="30000" enableLookups="false" />
>
> Apache MPM conf:
>
> ServerLimit 100
>
> StartServers 30
> MaxClients 2500
> MinSpareThreads 50
> MaxSpareThreads 100
> ThreadsPerChild 25
> MaxRequestsPerChild 0
>
>
> mod_proxy_ajp conf:
>
>
> BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app01 ttl=30 max=2
> BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app02 ttl=30 max=2
> BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app03 ttl=30 max=2
>
>
> Apache specs:
>
> Server version: Apache/2.2.13 (Unix)
> Server built: Sep 29 2009 11:12:35
> Server's Module Magic Number: 20051115:23
> Server loaded: APR 1.3.8, APR-Util 1.3.9
> Compiled using: APR 1.3.8, APR-Util 1.3.9
> Architecture: 32-bit
> Server MPM: Worker
> threaded: yes (fixed thread count)
> forked: yes (variable process count)
>
>
> I've beeng testing different settings for mod_proxy_ajp, but I am not
> sure what is the exact relation between the max option for mod_proxy
> and ThreadsPerChild. Should I set max to 1? I understand I would have
> as many connections to each Tomcat box as
> max*(MaxClients/ThreadsPerChild), is this correct?
>
> BTW, is the total number of connections that mod_proxy_ajp will
> forward to Tomcat related with MaxThreads ? I don't fully understand
> how APR connector works in this case.
>
Hi.
No great specialist here, but I'll go by the numbers above and give you
my interpretation.

MaxClients 2500
on your webserver/balancer indicates how many requests you are willing
to accept and process at the same time, at the Apache level.
ThreadsPerChild 25
means that, if you ever reach these 2500 simultaneous requests, Apache
will create 100 children processes to handle them, since each child is
limited to 25 threads, and 2500 / 25 threads -> 100 children.
(Which seems to match the ServerLimit. Good.)

It is so with the mod_jk connector, and probably also with mod_proy_ajp,
that the connector will automatically calculate how many connections to
make to Tomcat, based on the number of threads/processes above.

At the Tomcat level, the AJP Connector says
maxThreads="130"
and you have 3 Tomcats, thus for a total maximum number of threads of
130 X 3 = 390.
One Tomcat thread processes one request at a time.
So the Tomcats, together, can process 390 requests at a time.

There seems to be a mismatch between the number of requests you accept
in Apache (2500), and the number of threads available to process them in
Tomcat (390). At the limit, you could have 390 requests being processed
in Tomcat, and 2110 waiting to get a free Tomcat thread.

That is probably what you are seeing with this log message :
> [Fri Oct 30 14:47:43 2009] [error] (70007)The timeout specified has
> expired: ajp_ilink_receive() can't receive header

In other words, probably Apache/mod_proy_ajp got tired of waiting.

On the other hand, you indicate that the poor Tomcats are already
struggling to cope with the load. Increasing their number of threads is
thus unlikely to help much.
So it looks like you need more Tomcats.
Or else reduce the MaxClients in Apache.
Or optimise your Tomcat webapps.


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: mod_proxy_ajp (2.2.13) + Tomcat 6.0.20 - APR does

Hi, first of all, thanks for your answer.

And about the error, I also thought that could be the problem, but the
thing is when that error is logged on Apache, the Tomcat instances are
NOT using their MaxThreads capabilities.

Another issue we have is that, when this problem appears, the
processing threads on Tomcat start taking longer responses than
regular usage.

Thanks again,
-Damian

On Sat, Oct 31, 2009 at 7:46 PM, Andr=E9 Warnier wrote:
> Damian Traverso wrote:
>>
>> Hi,
>> we have one Apache-2.2.13 running mod_proxy_ajp + mod_proxy_balancer,
>> connected to (3) Tomcat-6.0.20 instances under Fedora release 8. We
>> are experiencing some issues with high CPU load on the Tomcat side,
>> and Apache starts logging errors like these
>>
>> [Fri Oct 30 14:47:43 2009] [error] (70007)The timeout specified has
>> expired: ajp_ilink_receive() can't receive header
>> [Fri Oct 30 14:47:43 2009] [error] ajp_read_header: ajp_ilink_receive
>> failed
>> [Fri Oct 30 14:47:43 2009] [error] (120006)APR does not understand
>> this error code: proxy: read response failed from
>>
>> On Tomcat side we are using the APR Native Connector, with this settings=
:
>>
>> =A0 >> maxThreads=3D"130" backlog=3D"10" connectionTimeout=3D"30000"
>> keepAliveTimeout=3D"30000" enableLookups=3D"false" />
>>
>> Apache MPM conf:
>>
>> ServerLimit 100
>>
>> =A0 StartServers =A0 =A0 =A0 =A0 30
>> =A0 MaxClients =A0 =A0 =A0 =A0 2500
>> =A0 MinSpareThreads =A0 =A0 =A050
>> =A0 MaxSpareThreads =A0 =A0 100
>> =A0 ThreadsPerChild =A0 =A0 =A025
>> =A0 MaxRequestsPerChild =A0 0
>>
>>
>> mod_proxy_ajp conf:
>>
>>
>> =A0 BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=3Dapp01 ttl=3D30 max=
=3D2
>> =A0 BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=3Dapp02 ttl=3D30 max=
=3D2
>> =A0 BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=3Dapp03 ttl=3D30 max=
=3D2
>>
>>
>> Apache specs:
>>
>> Server version: Apache/2.2.13 (Unix)
>> Server built: =A0 Sep 29 2009 11:12:35
>> Server's Module Magic Number: 20051115:23
>> Server loaded: =A0APR 1.3.8, APR-Util 1.3.9
>> Compiled using: APR 1.3.8, APR-Util 1.3.9
>> Architecture: =A0 32-bit
>> Server MPM: =A0 =A0 Worker
>> =A0threaded: =A0 =A0 yes (fixed thread count)
>> =A0forked: =A0 =A0 yes (variable process count)
>>
>>
>> I've beeng testing different settings for mod_proxy_ajp, but I am not
>> sure what is the exact relation between the max option for mod_proxy
>> and ThreadsPerChild. Should I set max to 1? I understand I would have
>> as many connections to each Tomcat box as
>> max*(MaxClients/ThreadsPerChild), is this correct?
>>
>> BTW, is the total number of connections that mod_proxy_ajp will
>> forward to Tomcat related with MaxThreads ? I don't fully understand
>> how APR connector works in this case.
>>
> Hi.
> No great specialist here, but I'll go by the numbers above and give you m=
y
> interpretation.
>
> MaxClients 2500
> on your webserver/balancer indicates how many requests you are willing to
> accept and process at the same time, at the Apache level.
> ThreadsPerChild 25
> means that, if you ever reach these 2500 simultaneous requests, Apache wi=
ll
> create 100 children processes to handle them, since each child is limited=
to
> 25 threads, and 2500 / 25 threads -> 100 children.
> (Which seems to match the ServerLimit. Good.)
>
> It is so with the mod_jk connector, and probably also with mod_proy_ajp,
> that the connector will automatically calculate how many connections to m=
ake
> to Tomcat, based on the number of threads/processes above.
>
> At the Tomcat level, the AJP Connector says
> maxThreads=3D"130"
> and you have 3 Tomcats, thus for a total maximum number of threads of 130=
X
> 3 =3D 390.
> One Tomcat thread processes one request at a time.
> So the Tomcats, together, can process 390 requests at a time.
>
> There seems to be a mismatch between the number of requests you accept in
> Apache (2500), and the number of threads available to process them in Tom=
cat
> (390). At the limit, you could have 390 requests being processed in Tomca=
t,
> and 2110 waiting to get a free Tomcat thread.
>
> That is probably what you are seeing with this log message :
>> [Fri Oct 30 14:47:43 2009] [error] (70007)The timeout specified has
>> expired: ajp_ilink_receive() can't receive header
>
> In other words, probably Apache/mod_proy_ajp got tired of waiting.
>
> On the other hand, you indicate that the poor Tomcats are already struggl=
ing
> to cope with the load. =A0Increasing their number of threads is thus unli=
kely
> to help much.
> So it looks like you need more Tomcats.
> Or else reduce the MaxClients in Apache.
> Or optimise your Tomcat webapps.
>
>
> ------------------------------------------------------------ ---------
> The official User-To-User support forum of the Apache HTTP Server Project=
..
> See for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> =A0" =A0 from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

RE: mod_proxy_ajp (2.2.13) + Tomcat 6.0.20 - APR does not understand this error code: proxy: read re

-----Original Message-----
From: Damian Traverso [mailto:damian.traverso@vostu.com]=20
Sent: Sunday, November 01, 2009 4:59 PM
To: users@httpd.apache.org
Subject: Re: [users@httpd] mod_proxy_ajp (2.2.13) + Tomcat 6.0.20 - APR =
does not understand this error code: proxy: read response failed

Hi, first of all, thanks for your answer.

And about the error, I also thought that could be the problem, but the
thing is when that error is logged on Apache, the Tomcat instances are
NOT using their MaxThreads capabilities.

Another issue we have is that, when this problem appears, the
processing threads on Tomcat start taking longer responses than
regular usage.

Thanks again,
-Damian

On Sat, Oct 31, 2009 at 7:46 PM, André Warnier =
wrote:
> Damian Traverso wrote:
>>
>> Hi,
>> we have one Apache-2.2.13 running mod_proxy_ajp + mod_proxy_balancer,
>> connected to (3) Tomcat-6.0.20 instances under Fedora release 8. We
>> are experiencing some issues with high CPU load on the Tomcat side,
>> and Apache starts logging errors like these
>>
>> [Fri Oct 30 14:47:43 2009] [error] (70007)The timeout specified has
>> expired: ajp_ilink_receive() can't receive header
>> [Fri Oct 30 14:47:43 2009] [error] ajp_read_header: ajp_ilink_receive
>> failed
>> [Fri Oct 30 14:47:43 2009] [error] (120006)APR does not understand
>> this error code: proxy: read response failed from
>>
>> On Tomcat side we are using the APR Native Connector, with this =
settings:
>>
>> >> maxThreads=3D"130" backlog=3D"10" connectionTimeout=3D"30000"
>> keepAliveTimeout=3D"30000" enableLookups=3D"false" />
>>
>> Apache MPM conf:
>>
>> ServerLimit 100
>>
>> StartServers 30
>> MaxClients 2500
>> MinSpareThreads 50
>> MaxSpareThreads 100
>> ThreadsPerChild 25
>> MaxRequestsPerChild 0
>>
>>
>> mod_proxy_ajp conf:
>>
>>
>> BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=3Dapp01 ttl=3D30 =
max=3D2
>> BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=3Dapp02 ttl=3D30 =
max=3D2
>> BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=3Dapp03 ttl=3D30 =
max=3D2
>>
>>
>> Apache specs:
>>
>> Server version: Apache/2.2.13 (Unix)
>> Server built: Sep 29 2009 11:12:35
>> Server's Module Magic Number: 20051115:23
>> Server loaded: APR 1.3.8, APR-Util 1.3.9
>> Compiled using: APR 1.3.8, APR-Util 1.3.9
>> Architecture: 32-bit
>> Server MPM: Worker
>> threaded: yes (fixed thread count)
>> forked: yes (variable process count)
>>
>>
>> I've beeng testing different settings for mod_proxy_ajp, but I am not
>> sure what is the exact relation between the max option for mod_proxy
>> and ThreadsPerChild. Should I set max to 1? I understand I would have
>> as many connections to each Tomcat box as
>> max*(MaxClients/ThreadsPerChild), is this correct?
>>
>> BTW, is the total number of connections that mod_proxy_ajp will
>> forward to Tomcat related with MaxThreads ? I don't fully understand
>> how APR connector works in this case.
>>
> Hi.
> No great specialist here, but I'll go by the numbers above and give =
you my
> interpretation.
>
> MaxClients 2500
> on your webserver/balancer indicates how many requests you are willing =
to
> accept and process at the same time, at the Apache level.
> ThreadsPerChild 25
> means that, if you ever reach these 2500 simultaneous requests, Apache =
will
> create 100 children processes to handle them, since each child is =
limited to
> 25 threads, and 2500 / 25 threads -> 100 children.
> (Which seems to match the ServerLimit. Good.)
>
> It is so with the mod_jk connector, and probably also with =
mod_proy_ajp,
> that the connector will automatically calculate how many connections =
to make
> to Tomcat, based on the number of threads/processes above.
>
> At the Tomcat level, the AJP Connector says
> maxThreads=3D"130"
> and you have 3 Tomcats, thus for a total maximum number of threads of =
130 X
> 3 =3D 390.
> One Tomcat thread processes one request at a time.
> So the Tomcats, together, can process 390 requests at a time.
>
> There seems to be a mismatch between the number of requests you accept =
in
> Apache (2500), and the number of threads available to process them in =
Tomcat
> (390). At the limit, you could have 390 requests being processed in =
Tomcat,
> and 2110 waiting to get a free Tomcat thread.
>
> That is probably what you are seeing with this log message :
>> [Fri Oct 30 14:47:43 2009] [error] (70007)The timeout specified has
>> expired: ajp_ilink_receive() can't receive header
>
> In other words, probably Apache/mod_proy_ajp got tired of waiting.
>
> On the other hand, you indicate that the poor Tomcats are already =
struggling
> to cope with the load. Increasing their number of threads is thus =
unlikely
> to help much.
> So it looks like you need more Tomcats.
> Or else reduce the MaxClients in Apache.
> Or optimise your Tomcat webapps.
>
>
> ------------------------------------------------------------ ---------
> The official User-To-User support forum of the Apache HTTP Server =
Project.
> See for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>

Hi Damian,

I've have experienced these errors for years over many versions of =
tomcat and APR/mod_ajp, always without a full load on Tomcat. I suspect =
it's a native socket management issue in the APR lib. I have noticed =
more errors on some OSs than others. But I'm no expert either.

What I do recommend is to stop using AJP and use mod_proxy_http instead. =
Unless you need the high degree of tunability of the AJP connectors or =
the SSL pass through features, IMHO HTTP proxying will give you less =
headaches and great performance under most loads. I used to have =
frequent errors, delays and even hanging servers with AJP which I could =
never diagnose effectively, but have no problems using HTTP proxying.

Also, if you apaches are serving your static content, then configuring =
more workers at the apache level than Tomcat threads is normal. In =
fact, if you are serving all your static content from Apache, depending =
on your hardware, then you may already have too _many_ threads =
configured on Tomcat. If there are some longer running requests to =
process which use the app server CPU intensively, you could easily swamp =
you Tomcat trying to process 130 simultaneous requests. If you've =
already load tested and this is the right setup for you workload then =
great.

Cheers,
Ryan


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org