Questions on mod_proxy Apache 2.0.52

I'm looking for a bit of direction on a few issues related to mod_proxy
and the Apache 2.X series.

Currently I'm in charge of Apache on a large website running 1.3.26
apache. Over the past few years the department has had a proliferation of
technologies for back end servers. I have been using mod_proxy to manage
connectivity to all of the servers in question. Unfortunately in the last
year our server has had issues handling the load restraints and buggy
application servers. To alleviate the immediate issues I made customized
changes to the 1.3.26 Apache mod_proxy to handle various timeout
conditions and real time monitoring. We also have been given the funding
to deploy a robust front end web server machine (An IBM blade center).

Given the magnitude of the migration to the new machine, I was given the
go ahead to investigate Apache 2.X as a better solution for the front end
web server. Notably of importance to us was the fact that a proxy timeout
directive was built into 2.X, and we would not need to worry about the
custom code. The load balancing failover that is being talked about will
also be of significant interest to us as it becomes available. I have run
into three issues, however.

1) SSL proxying. Due to security policies, we have a number of back end
app servers that require SSL from the client to the server. Therefore SSL
based proxying is a requirement. I have never seen a definitive statement
as to whether SSL proxying is supported, but I've seen indications it is
not, and confirmed in my tests that it did not work. Is there any plans to
implement this feature?

2) Timeout Directive. I tried using this with the test suite that I used
for my mod_proxy changes, and did not get the intended results. For
example, I wrote a cgi that wais 30 seconds before passing back a
response, and set the timeout to 10 seconds. On my version the proxy would
give up after 10 seconds of no data transfer. This did not happen with the
2.X timeout. Can someone give a better explanation of what this timeout
handles and whether I possibly made a configuration mistake.

3) Monitoring. My proxy changes wrote out a customize log entry upon
failure. I then wrote a program which analyzed this log in real time and
sent out warnings on configurable intervals when configurable thresholds
were breached. Assuming I can get 1 and 2 sorted out, I'd be willing to
work on this third item as an enhancement to mod_proxy.


Thanks


Jeffrey Burgoyne

Chief Technology Architect
KCSI Keenuh Consulting Services Inc
burgoyne@keenuh.com

Re: Questions on mod_proxy Apache 2.0.52

This is a cryptographically signed message in MIME format.

--------------ms030608020709010008050706
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

Jeffrey Burgoyne wrote:

> 1) SSL proxying. Due to security policies, we have a number of back end
> app servers that require SSL from the client to the server. Therefore SSL
> based proxying is a requirement. I have never seen a definitive statement
> as to whether SSL proxying is supported, but I've seen indications it is
> not, and confirmed in my tests that it did not work. Is there any plans to
> implement this feature?

If it doesn't work now, it is definitely desirable to make it work.

The v2.0 mod_proxy talks to the backend servers using the standard
filter stack, so making it talk SSL to the backend should be as
straightforward as adding the right filters to the stack at the right
time under the right circumstances.

If you're willing to submit code for this, I will definitely support
getting this into v2.0 (not only v2.2).

> 2) Timeout Directive. I tried using this with the test suite that I used
> for my mod_proxy changes, and did not get the intended results. For
> example, I wrote a cgi that wais 30 seconds before passing back a
> response, and set the timeout to 10 seconds. On my version the proxy would
> give up after 10 seconds of no data transfer. This did not happen with the
> 2.X timeout. Can someone give a better explanation of what this timeout
> handles and whether I possibly made a configuration mistake.

As I recall, the timeout directive handles the timeout after a
connection has been established - this definitely would need to be
looked at if it's not working properly.

> 3) Monitoring. My proxy changes wrote out a customize log entry upon
> failure. I then wrote a program which analyzed this log in real time and
> sent out warnings on configurable intervals when configurable thresholds
> were breached. Assuming I can get 1 and 2 sorted out, I'd be willing to
> work on this third item as an enhancement to mod_proxy.

This is also something really useful - please submit the patch :)

Regards,
Graham
--

--------------ms030608020709010008050706
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEH AQAAoIIJGzCC
AugwggJRoAMCAQICAwyZ8DANBgkqhkiG9w0BAQQFADBiMQswCQYDVQQGEwJa QTElMCMGA1UE
ChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhh d3RlIFBlcnNv
bmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwHhcNMDQwNjMwMTUxNjQ1WhcNMDUw NjMwMTUxNjQ1
WjBdMRAwDgYDVQQEEwdMZWdnZXR0MQ8wDQYDVQQqEwZHcmFoYW0xFzAVBgNV BAMTDkdyYWhh
bSBMZWdnZXR0MR8wHQYJKoZIhvcNAQkBFhBtaW5mcmluQHNoYXJwLmZtMIIB IjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwbwE90xkX5511UvMm4pwnFvv0nIIORsm +b+7Vgf04cob
H+fQaDVSDgKfZBm4lgoKQtv/2N+jXxzKtubau6yNMYvN+7iVkQJuLIjpo4DQ 2tb+hIvVsFvc
WkkFpm2+a8lIop1grh2OVIfxHfI/3OA4LbX1Ryq2qAou7TzQh6Te8KjdSigb f1l2gAyCT4ex
wLosSdHcTzv2WrYePJP107czC9gE237E68b+63Wmrc42Q4toz09XAaJnxebq SXWKhSx4h8cv
10hweAYXF5WiEUbINGoRD3V7pWRTbOBcz/oPpD8kh6kSu7iyDuchdOfIpy15 0ff/FCtI8h7f
LEXnBvh16wIDAQABoy0wKzAbBgNVHREEFDASgRBtaW5mcmluQHNoYXJwLmZt MAwGA1UdEwEB
/wQCMAAwDQYJKoZIhvcNAQEEBQADgYEAgTOjVmbVAi4gtKNhUI2UcMWE56z6 nG7KxQZ2EmJS
IDhXopbZsXtuOugBDxI1X49aqyQqOktHgWjiii/G0poKhNei3IrUuPB2bp9z o8MtiyB2brXg
lvj5N90jsA94MEMtnDLcdlP4C+XkyzarbUAh9TJxxmleateHTyZWIOZcPR0w ggLoMIICUaAD
AgECAgMMmfAwDQYJKoZIhvcNAQEEBQAwYjELMAkGA1UEBhMCWkExJTAjBgNV BAoTHFRoYXd0
ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJz b25hbCBGcmVl
bWFpbCBJc3N1aW5nIENBMB4XDTA0MDYzMDE1MTY0NVoXDTA1MDYzMDE1MTY0 NVowXTEQMA4G
A1UEBBMHTGVnZ2V0dDEPMA0GA1UEKhMGR3JhaGFtMRcwFQYDVQQDEw5HcmFo YW0gTGVnZ2V0
dDEfMB0GCSqGSIb3DQEJARYQbWluZnJpbkBzaGFycC5mbTCCASIwDQYJKoZI hvcNAQEBBQAD
ggEPADCCAQoCggEBAMG8BPdMZF+eddVLzJuKcJxb79JyCDkbJvm/u1YH9OHK Gx/n0Gg1Ug4C
n2QZuJYKCkLb/9jfo18cyrbm2rusjTGLzfu4lZECbiyI6aOA0NrW/oSL1bBb 3FpJBaZtvmvJ
SKKdYK4djlSH8R3yP9zgOC219UcqtqgKLu080Iek3vCo3UooG39ZdoAMgk+H scC6LEnR3E87
9lq2HjyT9dO3MwvYBNt+xOvG/ut1pq3ONkOLaM9PVwGiZ8Xm6kl1ioUseIfH L9dIcHgGFxeV
ohFGyDRqEQ91e6VkU2zgXM/6D6Q/JIepEru4sg7nIXTnyKctedH3/xQrSPIe 3yxF5wb4desC
AwEAAaMtMCswGwYDVR0RBBQwEoEQbWluZnJpbkBzaGFycC5mbTAMBgNVHRMB Af8EAjAAMA0G
CSqGSIb3DQEBBAUAA4GBAIEzo1Zm1QIuILSjYVCNlHDFhOes+pxuysUGdhJi UiA4V6KW2bF7
bjroAQ8SNV+PWqskKjpLR4Fo4oovxtKaCoTXotyK1Ljwdm6fc6PDLYsgdm61 4Jb4+TfdI7AP
eDBDLZwy3HZT+Avl5Ms2q21AIfUyccZpXmrXh08mViDmXD0dMIIDPzCCAqig AwIBAgIBDTAN
BgkqhkiG9w0BAQUFADCB0TELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rl cm4gQ2FwZTES
MBAGA1UEBxMJQ2FwZSBUb3duMRowGAYDVQQKExFUaGF3dGUgQ29uc3VsdGlu ZzEoMCYGA1UE
CxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEkMCIGA1UEAxMb VGhhd3RlIFBl
cnNvbmFsIEZyZWVtYWlsIENBMSswKQYJKoZIhvcNAQkBFhxwZXJzb25hbC1m cmVlbWFpbEB0
aGF3dGUuY29tMB4XDTAzMDcxNzAwMDAwMFoXDTEzMDcxNjIzNTk1OVowYjEL MAkGA1UEBhMC
WkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAq BgNVBAMTI1Ro
YXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMIGfMA0GCSqGSIb3 DQEBAQUAA4GN
ADCBiQKBgQDEpjxVc1X7TrnKmVoeaMB1BHCd3+n/ox7svc31W/Iadr1/DDph 8r9RzgHU5VAK
MNcCY1osiRVwjt3J8CuFWqo/cVbLrzwLB+fxH5E2JCoTzyvV84J3PQO+K/67 GD4Hv0CAAmTX
p6a7n2XRxSpUhQ9IBH+nttE8YQRAHmQZcmC3+wIDAQABo4GUMIGRMBIGA1Ud EwEB/wQIMAYB
Af8CAQAwQwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL2NybC50aGF3dGUuY29t L1RoYXd0ZVBl
cnNvbmFsRnJlZW1haWxDQS5jcmwwCwYDVR0PBAQDAgEGMCkGA1UdEQQiMCCk HjAcMRowGAYD
VQQDExFQcml2YXRlTGFiZWwyLTEzODANBgkqhkiG9w0BAQUFAAOBgQBIjNFQ g+oLLswNo2as
Zw9/r6y+whehQ5aUnX9MIbj4Nh+qLZ82L8D0HFAgk3A8/a3hYWLD2ToZfoSx mRsAxRoLgnSe
JVCUYsfbJ3FXJY3dqZw5jowgT2Vfldr394fWxghOrvbqNOUQGls1TXfjViF4 gtwhGTXeJLHT
HUb/XV9lTzGCAzswggM3AgEBMGkwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoT HFRoYXd0ZSBD
b25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25h bCBGcmVlbWFp
bCBJc3N1aW5nIENBAgMMmfAwCQYFKw4DAhoFAKCCAacwGAYJKoZIhvcNAQkD MQsGCSqGSIb3
DQEHATAcBgkqhkiG9w0BCQUxDxcNMDQxMDEzMTQyMzI5WjAjBgkqhkiG9w0B CQQxFgQUbYr6
YpN4lZva9zJbR3hoERaXuO8wUgYJKoZIhvcNAQkPMUUwQzAKBggqhkiG9w0D BzAOBggqhkiG
9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwIC ASgweAYJKwYB
BAGCNxAEMWswaTBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENv bnN1bHRpbmcg
KFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWls IElzc3Vpbmcg
Q0ECAwyZ8DB6BgsqhkiG9w0BCRACCzFroGkwYjELMAkGA1UEBhMCWkExJTAj BgNVBAoTHFRo
YXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQ ZXJzb25hbCBG
cmVlbWFpbCBJc3N1aW5nIENBAgMMmfAwDQYJKoZIhvcNAQEBBQAEggEAFTVr uZBwW+8vv4qm
6BzA1yLkTt5BeyzT8Golsp0NFzIBxzfSKeD5Co+V6xl/7XlAVpU3NgEfQMNi iOzo9RdB2zOU
jugG9pJNgUjHnDZCiQXS2a2IHX97A3BEPoIoz/ib16f/Z/ZykKhnMwDL3iPT 03AAblVc9fY7
iMU+5kNRIUovRDhzO7XIkb8a3si409ipJARc4GxDIrutNwDWQJorqODIvxwc S33NQ+CEvkp3
EQMp2Lbcav3lhMKmdXhfb+dunSguzODzdTILUoq9flmK34GbHKwsRJq5g4Tl LDfWSZDFbyKw
jQ60mVnHxF9gdj3c1tUZTg7cW2ck6zoDFbtecQAAAAAAAA==
--------------ms030608020709010008050706--

RE: Questions on mod_proxy Apache 2.0.52

As an Apache2/mod_proxy user (not developer), I can speak for the following:

> 1) SSL proxying. Due to security policies, we have a number
> of back end app servers that require SSL from the client to
> the server. Therefore SSL based proxying is a requirement. I
> have never seen a definitive statement as to whether SSL
> proxying is supported, but I've seen indications it is not,
> and confirmed in my tests that it did not work. Is there any
> plans to implement this feature?

We're using Apache2 mod_proxy as a reverse proxy with mod_ssl enabled,
proxying to a back-end WWW server over https. My understanding is
that the proxy is an SSL termination point, and it then opens a
new SSL connection to the back-end. You need to install certs on
both Apache and the back-end (although they can be the exact
same cert.) You also need to specify the "SSLProxyEngine On" directive
in your httpd.conf file.

A fact that you should also be aware of is that, technically speaking,
the request/response are being decrypted and re-encrypted in the
Apache mod_proxy process before being proxied onward to network.
Theoretically, this exposes you to man-in-the-middle issues... so good
host security and the latest patches are essential.

cheers
Eric

Re: Questions on mod_proxy Apache 2.0.52

On Wed, Oct 13, 2004 at 10:07:32AM -0400, Jeffrey Burgoyne wrote:
>
> 1) SSL proxying. Due to security policies, we have a number of back end
> app servers that require SSL from the client to the server. Therefore SSL
> based proxying is a requirement. I have never seen a definitive statement
> as to whether SSL proxying is supported, but I've seen indications it is
> not, and confirmed in my tests that it did not work. Is there any plans to
> implement this feature?

Hi,
We are using ssl mod_proxy since more than 2 years, you just need to use
ip based virtual host and the certificate in the reverse proxy. that's
all I think.

regards,
cahya.

Re: Questions on mod_proxy Apache 2.0.52

Graham, thanks for the reply. See comments below.

Jeffrey Burgoyne

Chief Technology Architect
KCSI Keenuh Consulting Services Inc
burgoyne@keenuh.com

On Wed, 13 Oct 2004, Graham Leggett wrote:

> Jeffrey Burgoyne wrote:
>
> > 1) SSL proxying. Due to security policies, we have a number of back end
> > app servers that require SSL from the client to the server. Therefore SSL
> > based proxying is a requirement. I have never seen a definitive statement
> > as to whether SSL proxying is supported, but I've seen indications it is
> > not, and confirmed in my tests that it did not work. Is there any plans to
> > implement this feature?
>
> If it doesn't work now, it is definitely desirable to make it work.
>
> The v2.0 mod_proxy talks to the backend servers using the standard
> filter stack, so making it talk SSL to the backend should be as
> straightforward as adding the right filters to the stack at the right
> time under the right circumstances.
>
> If you're willing to submit code for this, I will definitely support
> getting this into v2.0 (not only v2.2).

I may try to do this, but unfortunately due to time constraints I cannot
commit to working on this as a priority. From your explanation it does
look easier than I would have imagined, so I will try to take a cursory
look.

>
> > 2) Timeout Directive. I tried using this with the test suite that I used
> > for my mod_proxy changes, and did not get the intended results. For
> > example, I wrote a cgi that wais 30 seconds before passing back a
> > response, and set the timeout to 10 seconds. On my version the proxy would
> > give up after 10 seconds of no data transfer. This did not happen with the
> > 2.X timeout. Can someone give a better explanation of what this timeout
> > handles and whether I possibly made a configuration mistake.
>
> As I recall, the timeout directive handles the timeout after a
> connection has been established - this definitely would need to be
> looked at if it's not working properly.

Perhaps I configured it wrong. I have :

ProxyPass /cgi-bin/ http://strategis.ic.gc.ca/cgi-bin/
ProxyTimeout 10


When accessing /cgi-bin/timeoutin35seconds.cgi I immediately get the
connection as expected. The CGI though will not do anythign for 35 seconds
before sending back an HTML page. This did not timeout. Was I missing
something?

>
> > 3) Monitoring. My proxy changes wrote out a customize log entry upon
> > failure. I then wrote a program which analyzed this log in real time and
> > sent out warnings on configurable intervals when configurable thresholds
> > were breached. Assuming I can get 1 and 2 sorted out, I'd be willing to
> > work on this third item as an enhancement to mod_proxy.
>
> This is also something really useful - please submit the patch :)
>

I looked at this and will pretty much have to start from scratch. The way
did the proxy timeout in 1.3.26 is too different change to make my change
work. I did it by creating my own timeout function which took a the
seconds for a timeout followed by the URL being accessed. I then stored
the back end server socket in a global and used the callback and alarm.
Something like this :

(In http_main.c)

static void proxy_timeout(int sig)
{
strat_cancel_proxy_connection((request_rec *) timeout_req);
timeout(sig);
}


API_EXPORT(void) ap_config_timeout(int timeout_period, char *name,
request_rec *r)
{
#ifdef NETWARE
get_tsd
#endif
timeout_req = r;
timeout_name = name;

if (timeout_period < 1)
{
timeout_period = r->server->timeout;
}
ap_set_callback_and_alarm(proxy_timeout, timeout_period);
}

And in mod_proxy.c

void
strat_cancel_proxy_connection(request_rec *the_request)
{
extern int proxy_server_socket;

strat_log_proxy_failure(the_request->unparsed_uri);

if (proxy_server_socket != 0)
{
close(proxy_server_socket);
}

the_request->status = HTTP_BAD_GATEWAY;
ap_proxyerror(the_request, HTTP_BAD_GATEWAY, "Upstream Server
Unavailable.");
ap_send_error_response(the_request, 0);

return;
}


Teh changes made for the 2.0 were using the socket timeout I believe,
which is a more generic timeout function with its own function called upon
timeout.


So I will take a look at the logging, but for the most part will not be
able to reuse what I have done. I willbe able to reuse the monitroing
program though, as it is totally outside of Apache.


Thanks again.

Jeff


> Regards,
> Graham
> --
>

RE: Questions on mod_proxy Apache 2.0.52

Jeffrey Burgoyne

Chief Technology Architect
KCSI Keenuh Consulting Services Inc
burgoyne@keenuh.com

On Wed, 13 Oct 2004, Eric J. Hansen wrote:

> As an Apache2/mod_proxy user (not developer), I can speak for the following:
>
> > 1) SSL proxying. Due to security policies, we have a number
> > of back end app servers that require SSL from the client to
> > the server. Therefore SSL based proxying is a requirement. I
> > have never seen a definitive statement as to whether SSL
> > proxying is supported, but I've seen indications it is not,
> > and confirmed in my tests that it did not work. Is there any
> > plans to implement this feature?
>
> We're using Apache2 mod_proxy as a reverse proxy with mod_ssl enabled,
> proxying to a back-end WWW server over https. My understanding is
> that the proxy is an SSL termination point, and it then opens a
> new SSL connection to the back-end. You need to install certs on
> both Apache and the back-end (although they can be the exact
> same cert.) You also need to specify the "SSLProxyEngine On" directive
> in your httpd.conf file.

Thanks, I think that is it. Coming from the 1.3 world I simply used an
https in the url for the reverse.

When looking at the web site, I thought all the proxy directives were
under http://httpd.apache.org/docs-2.0/mod/mod_proxy.html . I missed the
comment on the third paragraph of the summary. Perhaps that section should
be a bit more prominent.



>
> A fact that you should also be aware of is that, technically speaking,
> the request/response are being decrypted and re-encrypted in the
> Apache mod_proxy process before being proxied onward to network.
> Theoretically, this exposes you to man-in-the-middle issues... so good
> host security and the latest patches are essential.

Absolutely. In fact, most people where I am on contract think security is
way overblown. We keep the whole environment very tightly locked down.


>
> cheers
> Eric
>
>

Re: Questions on mod_proxy Apache 2.0.52

Jeffrey Burgoyne

Chief Technology Architect
KCSI Keenuh Consulting Services Inc
burgoyne@keenuh.com

On Wed, 13 Oct 2004, Graham Leggett wrote:

> > 1) SSL proxying.
>


This is now working.

> > 2) Timeout Directive.

This one is still confounded me. I added in some debugging statements into
the code. For the startup phase, the value is definitely set properly.
When the proxy request is made, however, the set_timeout seems to be set
to 0 and hence the server timeout is then used.

I'll keep looking at this one and try to figure out exactly where the
value gets blanked. If someone wants to verify I've set up :

strategis.ic.gc.ca/cgi-bin/utils/timeout.cgi

is set up to hold the connection open, but return no data. Be warned the
firewall on our side will forcefully close this connection after 40
seconds, so try a lower timeout value.


>
> > 3) Monitoring.

I have a December/January timeframe for delivery of the project, so I'll
be looking at trying this sometime within the next few months as time
allows.

Thanks for the help everyone.

Jeff

Re: Questions on mod_proxy Apache 2.0.52

Jeffrey Burgoyne

Chief Technology Architect
KCSI Keenuh Consulting Services Inc
burgoyne@keenuh.com

On Wed, 13 Oct 2004, Jeffrey Burgoyne wrote:

> > > 2) Timeout Directive.
>
> This one is still confounded me. I added in some debugging statements into
> the code. For the startup phase, the value is definitely set properly.
> When the proxy request is made, however, the set_timeout seems to be set
> to 0 and hence the server timeout is then used.
>

Interesting results. I've trapped all places were the timeout_set variable
is set, and it seems to be 0 upon startup (default), set to 1 when the
ProxyTimeout is made, set to 1 on the merge config, yet 0 in the
ap_proxy_connect_to_backend function.

I have verified that the value placed into the timeout itself
(conf->timeout) is indeed the value that was set when set_proxy_timeout is
called. Exactly how the timeout_set is getting set to 0 is the mystery.
I'll keep looking.

Jeff