"proxy_balancer" | stickysession

Dear mailing list,

currently, we use the Apache2 version:
- Apache2 2.2.16 64bit
- standard "mod_proxy_balancer" module

with the following V-HOST configuration including a proxy balancer directiv=
e (with two Apache Tomcat instances behind):

BalancerMember ajp://rb-wcmstc1.xx.xxxxx.xxx:8009 loadfactor=3D100 retr=
y=3D10 route=3Drb-wcmstc1
BalancerMember ajp://rb-wcmstc2.xx.xxxxx.xxx:8009 loadfactor=3D100 retr=
y=3D10 route=3Drb-wcmstc2

ProxySet stickysession=3DJSESSIONID
ProxySet lbmethod=3Dbyrequests
#ProxySet nofailover=3DOn
ProxySet timeout=3D30


In production environment, we discover connection switches from "rb-wcmstc1=
" to "rb-wcmstc2" and vice versa (documented in the Tomcat-Logs) for the sa=
me Session-ID:

"rb-wcmstc1"
tomcat@:/opt/tomcat/logs $ grep -i "5893975846599935313" *
log4j-catalina.log.13:TRACE 29.09.2010 14:16:58 (de.espirit.firstspirit.io.=
servlet.ClientIOServlet): Opening connection to 10.35.32.123, id=3D58939758=
46599935313
log4j-catalina.log.13:TRACE 29.09.2010 14:16:58 (de.espirit.firstspirit.io.=
servlet.ClientIOServlet): Server call from 10.35.32.123, id=3D5893975846599=
935313, clientLength=3D82, serverLength=3D79, serverResponseLength=3D1, 455=
ms

"rb-wcmstc2"
DEBUG 29.09.2010 14:16:58 (de.espirit.firstspirit.io.servlet.ClientIOServle=
t): IO error with 10.35.32.123, port=3D1088, host=3Drb-wcmsfs4.de.bosch.com=
- java.
lang.IllegalStateException: Connection '5893975846599935313' not found
java.lang.IllegalStateException: Connection '5893975846599935313' not found
at de.espirit.firstspirit.io.servlet.ClientIOServlet.callServer (Cli=
entIOServlet.java:194)
at de.espirit.firstspirit.io.servlet.ClientIOServlet.doPost(Cli entI=
OServlet.java:117)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

Although, the "stickysession" attribute is set properly using the "JSESSION=
ID" cookie name and the Apache-Tomcat instances are up and running, the con=
nection SWITCHES. Why?

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: "proxy_balancer" | stickysession

Hi Holger,

On 09/29/2010 04:53 PM, King Holger (CI/AFP2) wrote:
> In production environment, we discover connection switches from
> "rb-wcmstc1" to "rb-wcmstc2" and vice versa (documented in the
> Tomcat-Logs) for the same Session-ID:
[...]
> Although, the "stickysession" attribute is set properly using the
> "JSESSIONID" cookie name and the Apache-Tomcat instances are up and
> running, the connection SWITCHES. Why?

It looks like you did not set the jvmRoute-Attribute of the Engine tag
in your Tomcat configuration to "rb-wcmstc1" on the first and
"rb-wcmstc2" on the second server.

mod_proxy_ajp (used by mod_proxy_balancer) identifies the Tomcat-backend
holding the session by looking up a suffix added to the JSESSIONID
cookie value. You can easily check whether the suffix is set or not by
logging the session id in the Tomcat access log (%S) or by logging
Cookie and Set-Cookie HTTP headers (logpattern: %{Cookie}i and
%{Set-Cookie}o) in your httpd access log.

Regards,
Thomas

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

RE: "proxy_balancer" | stickysession

Hi Thomas,

I had to disappoint you, because the relevant excerpt of the Tomcat configu=
ration below shows "jvmRoute" setup properly on EACH Tomcat instance:

"rb-wcmstc1"
1">

"rb-wcmstc2"
2">

So, might it be a BUG in Apache2 2.2.16? Is it possible to identify the rea=
son, why that happens on Apache2-side? How?

Kind regards,
Holger


-----Original Message-----
From: Thomas [mailto:apache.org-lists@darkwhole.de]
Sent: Mittwoch, 29. September 2010 23:31
To: users@httpd.apache.org
Subject: Re: [users@httpd] "proxy_balancer" | stickysession

Hi Holger,

On 09/29/2010 04:53 PM, King Holger (CI/AFP2) wrote:
> In production environment, we discover connection switches from
> "rb-wcmstc1" to "rb-wcmstc2" and vice versa (documented in the
> Tomcat-Logs) for the same Session-ID:
[...]
> Although, the "stickysession" attribute is set properly using the
> "JSESSIONID" cookie name and the Apache-Tomcat instances are up and
> running, the connection SWITCHES. Why?

It looks like you did not set the jvmRoute-Attribute of the Engine tag
in your Tomcat configuration to "rb-wcmstc1" on the first and
"rb-wcmstc2" on the second server.

mod_proxy_ajp (used by mod_proxy_balancer) identifies the Tomcat-backend
holding the session by looking up a suffix added to the JSESSIONID
cookie value. You can easily check whether the suffix is set or not by
logging the session id in the Tomcat access log (%S) or by logging
Cookie and Set-Cookie HTTP headers (logpattern: %{Cookie}i and
%{Set-Cookie}o) in your httpd access log.

Regards,
Thomas

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: "proxy_balancer" | stickysession

On Thursday 30 September 2010 09:54:11 King Holger (CI/AFP2) wrote:

Not only top posting, but also no quoting characters. Please fix that!

> Hi Thomas,
>
> I had to disappoint you, because the relevant excerpt of the Tomcat
> configuration below shows "jvmRoute" setup properly on EACH Tomcat
> instance:
>
> "rb-wcmstc1"
>
>
> "rb-wcmstc2"
>
>
> So, might it be a BUG in Apache2 2.2.16? Is it possible to identify the
> reason, why that happens on Apache2-side? How?

Do as Thomas recommended and capture the cookies and session ids. The IDs you
have in your first message look a bit unusual for tomcat generated session
IDs.

Additionally, you can set the Apache loglevel to debug for the VirtualHost
that does the loadbalancing. mod_proxy will log details about selecting the
backend worker.

Rainer

>
> Kind regards,
> Holger
>
>
> -----Original Message-----
> From: Thomas [mailto:apache.org-lists@darkwhole.de]
> Sent: Mittwoch, 29. September 2010 23:31
> To: users@httpd.apache.org
> Subject: Re: [users@httpd] "proxy_balancer" | stickysession
>
> Hi Holger,
>
> On 09/29/2010 04:53 PM, King Holger (CI/AFP2) wrote:
> > In production environment, we discover connection switches from
> > "rb-wcmstc1" to "rb-wcmstc2" and vice versa (documented in the
>
> > Tomcat-Logs) for the same Session-ID:
> [...]
>
> > Although, the "stickysession" attribute is set properly using the
> > "JSESSIONID" cookie name and the Apache-Tomcat instances are up and
> > running, the connection SWITCHES. Why?
>
> It looks like you did not set the jvmRoute-Attribute of the Engine tag
> in your Tomcat configuration to "rb-wcmstc1" on the first and
> "rb-wcmstc2" on the second server.
>
> mod_proxy_ajp (used by mod_proxy_balancer) identifies the Tomcat-backend
> holding the session by looking up a suffix added to the JSESSIONID
> cookie value. You can easily check whether the suffix is set or not by
> logging the session id in the Tomcat access log (%S) or by logging
> Cookie and Set-Cookie HTTP headers (logpattern: %{Cookie}i and
> %{Set-Cookie}o) in your httpd access log.
>
> Regards,
> Thomas
>
> ------------------------------------------------------------ ---------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
> ------------------------------------------------------------ ---------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: "proxy_balancer" | stickysession

On 29.09.2010 16:53, King Holger (CI/AFP2) wrote:
> Dear mailing list,
>
> currently, we use the Apache2 version:
> - Apache2 2.2.16 64bit
> - standard "mod_proxy_balancer" module
>
> with the following V-HOST configuration including a proxy balancer directive (with two Apache Tomcat instances behind):
>
> BalancerMember ajp://rb-wcmstc1.xx.xxxxx.xxx:8009 loadfactor=100 retry=10 route=rb-wcmstc1
> BalancerMember ajp://rb-wcmstc2.xx.xxxxx.xxx:8009 loadfactor=100 retry=10 route=rb-wcmstc2
>
> ProxySet stickysession=JSESSIONID
> ProxySet lbmethod=byrequests
> #ProxySet nofailover=On
> ProxySet timeout=30
>
>
> In production environment, we discover connection switches from "rb-wcmstc1" to "rb-wcmstc2" and vice versa (documented in the Tomcat-Logs) for the same Session-ID:
>
> "rb-wcmstc1"
> tomcat@:/opt/tomcat/logs $ grep -i "5893975846599935313" *
> log4j-catalina.log.13:TRACE 29.09.2010 14:16:58 (de.espirit.firstspirit.io.servlet.ClientIOServlet): Opening connection to 10.35.32.123, id=5893975846599935313
> log4j-catalina.log.13:TRACE 29.09.2010 14:16:58 (de.espirit.firstspirit.io.servlet.ClientIOServlet): Server call from 10.35.32.123, id=5893975846599935313, clientLength=82, serverLength=79, serverResponseLength=1, 455 ms
>
> "rb-wcmstc2"
> DEBUG 29.09.2010 14:16:58 (de.espirit.firstspirit.io.servlet.ClientIOServlet): IO error with 10.35.32.123, port=1088, host=rb-wcmsfs4.de.bosch.com - java.
> lang.IllegalStateException: Connection '5893975846599935313' not found
> java.lang.IllegalStateException: Connection '5893975846599935313' not found
> at de.espirit.firstspirit.io.servlet.ClientIOServlet.callServer (ClientIOServlet.java:194)
> at de.espirit.firstspirit.io.servlet.ClientIOServlet.doPost(Cli entIOServlet.java:117)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>
> Although, the "stickysession" attribute is set properly using the "JSESSIONID" cookie name and the Apache-Tomcat instances are up and running, the connection SWITCHES. Why?

How does stickyness work?
=========================

First it can use a cookie value or a value encoded in a URL. Java
webapps usually use a cookie named JSESSIONID, but when the client
doesn't support cookies, or if cookie usage is disabled on e.g. Tomcat,
it switches to URL encoding. Tomcat sometimes uses URL encoding when it
doesn't yet know, whether the client will support the cookie. URL
encoded session info is appended using ";jsessionid=...". Note the lower
case in the name of the id.

Since mod_proxy_balancer checks for the URL encoding and the cookie
using case sensitive strings, you need to configure both using the syntax:

stickysession=JSESSIONID|jsessionid

This is just to make sure, that an occasional URL encoded session info
is found as well.

Now by default mod_proxy_balancer looks for the URL encoded session info
as a parameter appended using "?" or "&", so as part of the query
string. Java Enterprise appends it using ";". To add that character to
the search list, you have to also set "scolonpathdelim On". Again this
is only necessary for occasional URL encoded sessions.

Now that mod_proxy_balancer can find the needed tokens (i.e. the value
of the JSESSIONID cookie, resp. the ;jsessionid URL encoded session
info), how does it detect the backend it should send the request to?

If there is a dot (".") in the token (session id), it only uses the part
after the dot as the "route", if not it uses the complete token.

Now it knows the "route" where it should send the request to it looks
for a balancer member, which has the same value set as its "route". In
the above example "rb-wcmstc1" respectively "rb-wcmstc2". This member
will handle the request - if the member is available, i.e. not broken or
disabled.

Finally: in order to let Tomcat attach the needed route to the end of
the session id you need to set the jvmRoute attribute in server.xml to
the respective route value.


How to track failures
=====================

1) Check whether your Tomcats send the correct route in the session id.

Look at the JSESSIONID cookie in the browser, whether it has
".rb-wcmstc1" resp. ".rb-wcmstc2" attached at the end of the id value.


2) Log the session ids send by the clients, and the session id set by
the server.

Activate the Tomcat access log and change the pattern to include

"%S "%{Cookie}i" "%{Set-Cookie}o""

Change your log format for the Apache access log using CustomLog and add

"%{JSESSIONID}C" "%{Set-Cookie}o"

or even

"%{Cookie}i" "%{JSESSIONID}C" "%{Set-Cookie}o"

On the docs page

http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html #environment

you will also find a list of Apache environment variables useful for
debugging when adding to the log format, e.g.

%{BALANCER_SESSION_STICKY}e %{BALANCER_SESSION_ROUTE}e
%{BALANCER_WORKER_NAME}e %{BALANCER_WORKER_ROUTE}e
%{BALANCER_ROUTE_CHANGED}e


3) Switch Apache to debug log level

CAUTION: High log volume by mod_proxy and also mod_ssl if used.

It will log e.g.

proxy: BALANCER: Found value for stickysession

(that's the token contained in the request)

proxy: BALANCER: Found route

(that's the route info from the token, e.g. stripping everything in
front of a dot)

proxy: BALANCER: Route changed from to

(if there is no worker available for the route , the worker
which will be used instead is ).


Final remarks
=============

You look for the string "5893975846599935313" in your above log
snippets. That does not look like a Tomcat session id. It doesn't have
the trailing route and it is purely decimal numeric, whereas a Tomcat
session id is hexadecimal. It is very unlikely that there are no [A-F]
digits in it.

Regards,

Rainer

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org