mod_proxy_balancer back end status check

--_000_F41794BFF0B8CE4B9CF5C56209A983220253C80669PRDEXC103ig ii_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,

When using mod_proxy_balancer, is it possible to configure the way it check=
s the status of the member servers in its cluster?
I have come across a scenario where the back end server is running, and acc=
epts requests but doesn't send responses. This gives the impression it is u=
p, but in actual fact it's as good as down. It would be good if we could se=
t a timeout, or call a status script from mod_proxy_balancer, but I haven't=
been able to find anything about this in the documentation. Any ideas?

Thanks for any help.

Edward.

________________________________
The information contained in this email is strictly confidential and for th=
e use of the addressee only, unless otherwise indicated. If you are not the=
intended recipient, please do not read, copy, use or disclose to others th=
is message or any attachment. Please also notify the sender by replying to =
this email or by telephone (+44 (0)20 7896 0011) and then delete the email =
and any copies of it. Opinions, conclusions (etc.) that do not relate to th=
e official business of this company shall be understood as neither given no=
r endorsed by it. IG Index Ltd is a company registered in England and Wales=
under number 01190902. VAT registration number 761 2978 07. Registered Off=
ice: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and=
regulated by the Financial Services Authority. FSA Register number 114059.

--_000_F41794BFF0B8CE4B9CF5C56209A983220253C80669PRDEXC103ig ii_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">

>

RE: mod_proxy_balancer back end status check

--_000_F41794BFF0B8CE4B9CF5C56209A983220253C809CBPRDEXC103ig ii_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Reposting.

Here's my configuration. After the 15 seconds timeout, the back end server =
(listening on port 8080 here) returns a 502 page. Is there any configurati=
on I can add to detect that beforehand, and get the balancer to redirect my=
request to the other server member?

Timeout 15

ProxyPass /HelloWorld/ balancer://tomcat/HelloWorld/
ProxyPassReverse /HelloWorld/ balancer://tomcat/HelloWorld/


BalancerMember http://localhost:8080 route=3Djvm001 redirect=3Djvm002
BalancerMember http://localhost:8082 route=3Djvm002



From: Edward Quick [mailto:Edward.Quick@igindex.co.uk]
Sent: 08 March 2010 11:13
To: users@httpd.apache.org
Subject: [users@httpd] mod_proxy_balancer back end status check

Hi,

When using mod_proxy_balancer, is it possible to configure the way it check=
s the status of the member servers in its cluster?
I have come across a scenario where the back end server is running, and acc=
epts requests but doesn't send responses. This gives the impression it is u=
p, but in actual fact it's as good as down. It would be good if we could se=
t a timeout, or call a status script from mod_proxy_balancer, but I haven't=
been able to find anything about this in the documentation. Any ideas?

Thanks for any help.

Edward.

________________________________
The information contained in this email is strictly confidential and for th=
e use of the addressee only, unless otherwise indicated. If you are not the=
intended recipient, please do not read, copy, use or disclose to others th=
is message or any attachment. Please also notify the sender by replying to =
this email or by telephone (+44 (0)20 7896 0011) and then delete the email =
and any copies of it. Opinions, conclusions (etc.) that do not relate to th=
e official business of this company shall be understood as neither given no=
r endorsed by it. IG Index Ltd is a company registered in England and Wales=
under number 01190902. VAT registration number 761 2978 07. Registered Off=
ice: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and=
regulated by the Financial Services Authority. FSA Register number 114059.

________________________________
The information contained in this email is strictly confidential and for th=
e use of the addressee only, unless otherwise indicated. If you are not the=
intended recipient, please do not read, copy, use or disclose to others th=
is message or any attachment. Please also notify the sender by replying to =
this email or by telephone (+44 (0)20 7896 0011) and then delete the email =
and any copies of it. Opinions, conclusions (etc.) that do not relate to th=
e official business of this company shall be understood as neither given no=
r endorsed by it. IG Index Ltd is a company registered in England and Wales=
under number 01190902. VAT registration number 761 2978 07. Registered Off=
ice: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and=
regulated by the Financial Services Authority. FSA Register number 114059.

--_000_F41794BFF0B8CE4B9CF5C56209A983220253C809CBPRDEXC103ig ii_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:x=3D"urn:schemas-microsoft-com:office:excel" xmlns:p=3D"urn:schemas-m=
icrosoft-com:office:powerpoint" xmlns:a=3D"urn:schemas-microsoft-com:office=
:access" xmlns:dt=3D"uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s=3D"=
uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs=3D"urn:schemas-microsof=
t-com:rowset" xmlns:z=3D"#RowsetSchema" xmlns:b=3D"urn:schemas-microsoft-co=
m:office:publisher" xmlns:ss=3D"urn:schemas-microsoft-com:office:spreadshee=
t" xmlns:c=3D"urn:schemas-microsoft-com:office:component:spread sheet" xmlns=
:odc=3D"urn:schemas-microsoft-com:office:odc" xmlns:oa=3D"urn:schemas-micro=
soft-com:office:activation" xmlns:html=3D"http://www.w3.org/TR/REC-html40" =
xmlns:q=3D"http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc=3D"http://m=
icrosoft.com/officenet/conferencing" xmlns:D=3D"DAV:" xmlns:Repl=3D"http://=
schemas.microsoft.com/repl/" xmlns:mt=3D"http://schemas.microsoft.com/share=
point/soap/meetings/" xmlns:x2=3D"http://schemas.microsoft.com/office/excel=
/2003/xml" xmlns:ppda=3D"http://www.passport.com/NameSpace.xsd" xmlns:ois=
=3D"http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir=3D"http://=
schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds=3D"http://www.w3=
..org/2000/09/xmldsig#" xmlns:dsp=3D"http://schemas.microsoft.com/sharepoint=
/dsp" xmlns:udc=3D"http://schemas.microsoft.com/data/udc" xmlns:xsd=3D"http=
://www.w3.org/2001/XMLSchema" xmlns:sub=3D"http://schemas.microsoft.com/sha=
repoint/soap/2002/1/alerts/" xmlns:ec=3D"http://www.w3.org/2001/04/xmlenc#"=
xmlns:sp=3D"http://schemas.microsoft.com/sharepoint/" xmlns:sps=3D"http://=
schemas.microsoft.com/sharepoint/soap/" xmlns:xsi=3D"http://www.w3.org/2001=
/XMLSchema-instance" xmlns:udcs=3D"http://schemas.microsoft.com/data/udc/so=
ap" xmlns:udcxf=3D"http://schemas.microsoft.com/data/udc/xmlfile " xmlns:udc=
p2p=3D"http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf=3D"http:/=
/schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss=3D"http://sche=
mas.microsoft.com/office/2006/digsig-setup" xmlns:dssi=3D"http://schemas.mi=
crosoft.com/office/2006/digsig" xmlns:mdssi=3D"http://schemas.openxmlformat=
s.org/package/2006/digital-signature" xmlns:mver=3D"http://schemas.openxmlf=
ormats.org/markup-compatibility/2006" xmlns:m=3D"http://schemas.microsoft.c=
om/office/2004/12/omml" xmlns:mrels=3D"http://schemas.openxmlformats.org/pa=
ckage/2006/relationships" xmlns:spwp=3D"http://microsoft.com/sharepoint/web=
partpages" xmlns:ex12t=3D"http://schemas.microsoft.com/exchange/service s/20=
06/types" xmlns:ex12m=3D"http://schemas.microsoft.com/exchange/service s/200=
6/messages" xmlns:pptsl=3D"http://schemas.microsoft.com/sharepoint/soap/ Sli=
deLibrary/" xmlns:spsl=3D"http://microsoft.com/webservices/SharePointPor tal=
Server/PublishedLinksService" xmlns:Z=3D"urn:schemas-microsoft-com:" xmlns:=
st=3D"
" xmlns=3D"http://www.w3.org/TR/REC-html40">

>

Re: RE: mod_proxy_balancer back end status check

On 9 March 2010 17:47, Edward Quick wrote:
> Here=92s my configuration. After the 15 seconds timeout, the back end ser=
ver
> (listening on port 8080 here) =A0returns a 502 page. Is there any
> configuration I can add to detect that beforehand, and get the balancer t=
o
> redirect my request to the other server member?
>
> Timeout 15
>
> ProxyPass /HelloWorld/ balancer://tomcat/HelloWorld/
> ProxyPassReverse /HelloWorld/ balancer://tomcat/HelloWorld/
>
>
> =A0BalancerMember http://localhost:8080 route=3Djvm001 redirect=3Djvm002
> =A0BalancerMember http://localhost:8082 route=3Djvm002
>

You could try using AJP instead of HTTP between Apache and Tomcat, and
use the ping parameter. Search for 'ping' here:-

http://httpd.apache.org/docs/2.2/mod/mod_proxy.html

-- Phil

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

RE: RE: mod_proxy_balancer back end status check

Thanks Phil,

That worked a treat.


BalancerMember ajp://localhost:8009 route=3Djvm001 redirect=3Djvm002 keepa=
live=3DOn ping=3D1
BalancerMember ajp://localhost:8010 route=3Djvm002 keepalive=3DOn status=
=3D-H ping=3D1




-----Original Message-----
From: Philip Wigg [mailto:phil@philipwigg.co.uk]
Sent: 09 March 2010 19:57
To: users@httpd.apache.org
Subject: Re: [users@httpd] RE: mod_proxy_balancer back end status check

On 9 March 2010 17:47, Edward Quick wrote:
> Here's my configuration. After the 15 seconds timeout, the back end
> server (listening on port 8080 here) returns a 502 page. Is there any
> configuration I can add to detect that beforehand, and get the
> balancer to redirect my request to the other server member?
>
> Timeout 15
>
> ProxyPass /HelloWorld/ balancer://tomcat/HelloWorld/ ProxyPassReverse
> /HelloWorld/ balancer://tomcat/HelloWorld/
>
>
> BalancerMember http://localhost:8080 route=3Djvm001 redirect=3Djvm002
> BalancerMember http://localhost:8082 route=3Djvm002

You could try using AJP instead of HTTP between Apache and Tomcat, and use =
the ping parameter. Search for 'ping' here:-

http://httpd.apache.org/docs/2.2/mod/mod_proxy.html

-- Phil

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


The information contained in this email is strictly confidential and for th=
e use of the addressee only, unless otherwise indicated. If you are not the=
intended recipient, please do not read, copy, use or disclose to others th=
is message or any attachment. Please also notify the sender by replying to =
this email or by telephone (+44 (0)20 7896 0011) and then delete the email =
and any copies of it. Opinions, conclusions (etc.) that do not relate to th=
e official business of this company shall be understood as neither given no=
r endorsed by it. IG Index Ltd is a company registered in England and Wales=
under number 01190902. VAT registration number 761 2978 07. Registered Off=
ice: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and=
regulated by the Financial Services Authority. FSA Register number 114059.

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

RE: RE: mod_proxy_balancer back end status check

Further testing showed this succeeded when the server was busy. The balance=
r marked the member's status as Err.
However later after firing continuous requests, the backend ran out of memo=
ry, causing it to return a 500, and then the balancer marked it up again, s=
igh!

I'm considering writing a script to check the backend and disable the membe=
rs via a curl call to the balancer-manager page, but that seems a bit crud=
e. Is there anything else I can add to the configuration below which might =
handle a code 500?


-----Original Message-----
From: Edward Quick [mailto:Edward.Quick@igindex.co.uk]
Sent: 09 March 2010 22:08
To: users@httpd.apache.org
Subject: RE: [users@httpd] RE: mod_proxy_balancer back end status check

Thanks Phil,

That worked a treat.


BalancerMember ajp://localhost:8009 route=3Djvm001 redirect=3Djvm002 keepa=
live=3DOn ping=3D1 BalancerMember ajp://localhost:8010 route=3Djvm002 keep=
alive=3DOn status=3D-H ping=3D1



-----Original Message-----
From: Philip Wigg [mailto:phil@philipwigg.co.uk]
Sent: 09 March 2010 19:57
To: users@httpd.apache.org
Subject: Re: [users@httpd] RE: mod_proxy_balancer back end status check

On 9 March 2010 17:47, Edward Quick wrote:
> Here's my configuration. After the 15 seconds timeout, the back end
> server (listening on port 8080 here) returns a 502 page. Is there any
> configuration I can add to detect that beforehand, and get the
> balancer to redirect my request to the other server member?
>
> Timeout 15
>
> ProxyPass /HelloWorld/ balancer://tomcat/HelloWorld/ ProxyPassReverse
> /HelloWorld/ balancer://tomcat/HelloWorld/
>
>
> BalancerMember http://localhost:8080 route=3Djvm001 redirect=3Djvm002
> BalancerMember http://localhost:8082 route=3Djvm002

You could try using AJP instead of HTTP between Apache and Tomcat, and use =
the ping parameter. Search for 'ping' here:-

http://httpd.apache.org/docs/2.2/mod/mod_proxy.html

-- Phil

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


The information contained in this email is strictly confidential and for th=
e use of the addressee only, unless otherwise indicated. If you are not the=
intended recipient, please do not read, copy, use or disclose to others th=
is message or any attachment. Please also notify the sender by replying to =
this email or by telephone (+44 (0)20 7896 0011) and then delete the email =
and any copies of it. Opinions, conclusions (etc.) that do not relate to th=
e official business of this company shall be understood as neither given no=
r endorsed by it. IG Index Ltd is a company registered in England and Wales=
under number 01190902. VAT registration number 761 2978 07. Registered Off=
ice: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and=
regulated by the Financial Services Authority. FSA Register number 114059.

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


The information contained in this email is strictly confidential and for th=
e use of the addressee only, unless otherwise indicated. If you are not the=
intended recipient, please do not read, copy, use or disclose to others th=
is message or any attachment. Please also notify the sender by replying to =
this email or by telephone (+44 (0)20 7896 0011) and then delete the email =
and any copies of it. Opinions, conclusions (etc.) that do not relate to th=
e official business of this company shall be understood as neither given no=
r endorsed by it. IG Index Ltd is a company registered in England and Wales=
under number 01190902. VAT registration number 761 2978 07. Registered Off=
ice: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and=
regulated by the Financial Services Authority. FSA Register number 114059.

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: RE: mod_proxy_balancer back end status check

On 9 March 2010 23:01, Edward Quick wrote:
> Further testing showed this succeeded when the server was busy. The balan=
cer marked the member's status as Err.
> However later after firing continuous requests, the backend ran out of me=
mory, causing it to return a 500, and then the balancer marked it up again,=
sigh!
>
> I'm considering writing a script to check the backend and disable the mem=
bers via a curl call to the balancer-manager page, but that =A0seems a bit =
crude. Is there anything else I can add to the configuration below which mi=
ght handle a code 500?

Hmm. Unfortunately I don't know of anything else you could do to fix
that with mod_proxy. Someone else might have a way.

Are you sure the 500 is definitely being received from the back-end
and not from Apache/mod_proxy itself because it doesn't have anywhere
left to route requests to? I'm surprised Tomcat returns a cpong in
that scenario but I've never tried it.

Presuming it is Tomcat returning the 500, you might try:-

1. Make sure it is running out of heap that's taking down Tomcat and
not something else like running out of PermGen space or threads.
Presuming you've done this...

2. Fixing the app so it doesn't run out of heap. I appreciate if
you're the sysadmin that it can be difficult to get the devs to do
this.

2. Is the app in production currently? How long does it typically take
to run out of heap space? Can you just restart it once a day whilst
you encourage the devs to fix it? If not, can you increase the max
heap space until it lasts longer than a day? You can get stats from
Tomcat in XML format by accessing the manager page using
/manager/status?XML=3Dtrue which makes it easy to log stats you're
interested in using curl, grep, or perl.

3. Monitor your log files for JavaOutOfMemory errors (you should get
these if Tomcat is running out of heap) using something like
http://sourceforge.net/projects/swatch/ which allows you to take an
action if it sees that string. Use it to automatically restart Tomcat
in the meantime (whilst the devs are fixing the main issue). I've used
this successfully in the past when all else fails.

4. Do you only have one front-end web server? If you have multiple web
servers, do you have a hardware load-balancer in front? If so, tie
each Apache server to a single Tomcat instance and use the front-end
load-balancer to remove the web server from the load-balanced pool
once it starts throwing a 500. You'll need to make sure you always
have at least one working server Tomcat though by implementing
something like option 3.

The Tomcat mailing list would be best placed to answer questions about
going down the fixing-Tomcat road. Good luck fixing your issue.

-- Phil

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

RE: RE: mod_proxy_balancer back end status check

Phil,

Thanks for your response, and all the excellent suggestions.
I found a bug issue raised about this very problem https://issues.apache.or=
g/bugzilla/show_bug.cgi?id=3D47207

In my case I intentionally invoked the 500 error on the backend tomcat as I=
have seen these issues in the past, so
I was hoping to be able to handle this at the apache (load-balance) layer. =
However, it doesn't sound like this is possible at the moment.

Thanks again,

Ed.





-----Original Message-----
From: Philip Wigg [mailto:phil@philipwigg.co.uk]
Sent: 10 March 2010 09:28
To: users@httpd.apache.org
Subject: Re: [users@httpd] RE: mod_proxy_balancer back end status check

On 9 March 2010 23:01, Edward Quick wrote:
> Further testing showed this succeeded when the server was busy. The balan=
cer marked the member's status as Err.
> However later after firing continuous requests, the backend ran out of me=
mory, causing it to return a 500, and then the balancer marked it up again,=
sigh!
>
> I'm considering writing a script to check the backend and disable the mem=
bers via a curl call to the balancer-manager page, but that seems a bit cr=
ude. Is there anything else I can add to the configuration below which migh=
t handle a code 500?

Hmm. Unfortunately I don't know of anything else you could do to fix that w=
ith mod_proxy. Someone else might have a way.

Are you sure the 500 is definitely being received from the back-end and not=
from Apache/mod_proxy itself because it doesn't have anywhere left to rout=
e requests to? I'm surprised Tomcat returns a cpong in that scenario but I'=
ve never tried it.

Presuming it is Tomcat returning the 500, you might try:-

1. Make sure it is running out of heap that's taking down Tomcat and not so=
mething else like running out of PermGen space or threads.
Presuming you've done this...

2. Fixing the app so it doesn't run out of heap. I appreciate if you're the=
sysadmin that it can be difficult to get the devs to do this.

2. Is the app in production currently? How long does it typically take to r=
un out of heap space? Can you just restart it once a day whilst you encoura=
ge the devs to fix it? If not, can you increase the max heap space until it=
lasts longer than a day? You can get stats from Tomcat in XML format by ac=
cessing the manager page using /manager/status?XML=3Dtrue which makes it ea=
sy to log stats you're interested in using curl, grep, or perl.

3. Monitor your log files for JavaOutOfMemory errors (you should get these =
if Tomcat is running out of heap) using something like http://sourceforge.n=
et/projects/swatch/ which allows you to take an action if it sees that stri=
ng. Use it to automatically restart Tomcat in the meantime (whilst the devs=
are fixing the main issue). I've used this successfully in the past when a=
ll else fails.

4. Do you only have one front-end web server? If you have multiple web serv=
ers, do you have a hardware load-balancer in front? If so, tie each Apache =
server to a single Tomcat instance and use the front-end load-balancer to r=
emove the web server from the load-balanced pool once it starts throwing a =
500. You'll need to make sure you always have at least one working server T=
omcat though by implementing something like option 3.

The Tomcat mailing list would be best placed to answer questions about goin=
g down the fixing-Tomcat road. Good luck fixing your issue.

-- Phil

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


The information contained in this email is strictly confidential and for th=
e use of the addressee only, unless otherwise indicated. If you are not the=
intended recipient, please do not read, copy, use or disclose to others th=
is message or any attachment. Please also notify the sender by replying to =
this email or by telephone (+44 (0)20 7896 0011) and then delete the email =
and any copies of it. Opinions, conclusions (etc.) that do not relate to th=
e official business of this company shall be understood as neither given no=
r endorsed by it. IG Index Ltd is a company registered in England and Wales=
under number 01190902. VAT registration number 761 2978 07. Registered Off=
ice: Friars House, 157-168 Blackfriars Road, London SE1 8EZ. Authorised and=
regulated by the Financial Services Authority. FSA Register number 114059.

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org