mod_proxy reverse proxy optimization/performance question

I am using a reverse proxy to cache a remote site. The files are mostly
rpms, with varying sizes: 3-30M or more.
Now if you have a number of requests for the same file which is not yet
cached locally, all of these requests will download the requested file
from the remote site. It will slow down the speed of each download as
the throughput of the line will be split among all processes.
So if there are lots of processes to download the same rpm from a remote
site, this can take lots of time to complete a request.
This can bring apache to a state where it can not serve other requests,
as all available processes are already busy.


In my opinion it would be more efficient to let one process complete the
request (using maximum line throughput) and return some busy code to
other identical, simultaneous requests until the file is cached locally.
As anyone run into a similar situation? What solution did you find?

I have created a solution, as I did not find anything else already
existing. I would like to discuss it here and get your opinions.
1. When a request for a file that is not yet in the local cache is
accepted by the proxy, a temporary lock file is created (based on the
proxy's pathname of the file, changed from directory slashes to
underscores).
2. Other processes requesting the same file will check first for the
lock file. If found, they will return a busy code (ie: 408 Request
Timeout), and the request should be sent repeatedly until successful.

Please let me know what you think of this approach, especially if you
have done or seen something similar.

--
------------------------------------------------------------ -
I am root. If you see me laughing... You better have a backup!

Re: mod_proxy reverse proxy optimization/performance question

This is a cryptographically signed message in MIME format.

--------------ms070100040701070201050008
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

Roman Gavrilov wrote:

> I am using a reverse proxy to cache a remote site. The files are mostly
> rpms, with varying sizes: 3-30M or more.
> Now if you have a number of requests for the same file which is not yet
> cached locally, all of these requests will download the requested file
> from the remote site. It will slow down the speed of each download as
> the throughput of the line will be split among all processes.
> So if there are lots of processes to download the same rpm from a remote
> site, this can take lots of time to complete a request.
> This can bring apache to a state where it can not serve other requests,
> as all available processes are already busy.

This is a mod_cache issue rather than a proxy issue, the best place to
discuss something like this is dev@httpd.apache.org. (mod_cache was
separated from mod_proxy in httpd v2.0, this fix never went into httpd
v1.3 mod_proxy because it was a serious architecture change)

When mod_cache was separated from mod_proxy in httpd v2.0, one of the
problems the new cache code was supposed to solve was this exact problem
- whether this problem stayed solved in all the development to mod_cache
that has been done in the last while is a good question.

Regards,
Graham
--

--------------ms070100040701070201050008
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEH AQAAoIIJGzCC
AugwggJRoAMCAQICAwyZ8DANBgkqhkiG9w0BAQQFADBiMQswCQYDVQQGEwJa QTElMCMGA1UE
ChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhh d3RlIFBlcnNv
bmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwHhcNMDQwNjMwMTUxNjQ1WhcNMDUw NjMwMTUxNjQ1
WjBdMRAwDgYDVQQEEwdMZWdnZXR0MQ8wDQYDVQQqEwZHcmFoYW0xFzAVBgNV BAMTDkdyYWhh
bSBMZWdnZXR0MR8wHQYJKoZIhvcNAQkBFhBtaW5mcmluQHNoYXJwLmZtMIIB IjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwbwE90xkX5511UvMm4pwnFvv0nIIORsm +b+7Vgf04cob
H+fQaDVSDgKfZBm4lgoKQtv/2N+jXxzKtubau6yNMYvN+7iVkQJuLIjpo4DQ 2tb+hIvVsFvc
WkkFpm2+a8lIop1grh2OVIfxHfI/3OA4LbX1Ryq2qAou7TzQh6Te8KjdSigb f1l2gAyCT4ex
wLosSdHcTzv2WrYePJP107czC9gE237E68b+63Wmrc42Q4toz09XAaJnxebq SXWKhSx4h8cv
10hweAYXF5WiEUbINGoRD3V7pWRTbOBcz/oPpD8kh6kSu7iyDuchdOfIpy15 0ff/FCtI8h7f
LEXnBvh16wIDAQABoy0wKzAbBgNVHREEFDASgRBtaW5mcmluQHNoYXJwLmZt MAwGA1UdEwEB
/wQCMAAwDQYJKoZIhvcNAQEEBQADgYEAgTOjVmbVAi4gtKNhUI2UcMWE56z6 nG7KxQZ2EmJS
IDhXopbZsXtuOugBDxI1X49aqyQqOktHgWjiii/G0poKhNei3IrUuPB2bp9z o8MtiyB2brXg
lvj5N90jsA94MEMtnDLcdlP4C+XkyzarbUAh9TJxxmleateHTyZWIOZcPR0w ggLoMIICUaAD
AgECAgMMmfAwDQYJKoZIhvcNAQEEBQAwYjELMAkGA1UEBhMCWkExJTAjBgNV BAoTHFRoYXd0
ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJz b25hbCBGcmVl
bWFpbCBJc3N1aW5nIENBMB4XDTA0MDYzMDE1MTY0NVoXDTA1MDYzMDE1MTY0 NVowXTEQMA4G
A1UEBBMHTGVnZ2V0dDEPMA0GA1UEKhMGR3JhaGFtMRcwFQYDVQQDEw5HcmFo YW0gTGVnZ2V0
dDEfMB0GCSqGSIb3DQEJARYQbWluZnJpbkBzaGFycC5mbTCCASIwDQYJKoZI hvcNAQEBBQAD
ggEPADCCAQoCggEBAMG8BPdMZF+eddVLzJuKcJxb79JyCDkbJvm/u1YH9OHK Gx/n0Gg1Ug4C
n2QZuJYKCkLb/9jfo18cyrbm2rusjTGLzfu4lZECbiyI6aOA0NrW/oSL1bBb 3FpJBaZtvmvJ
SKKdYK4djlSH8R3yP9zgOC219UcqtqgKLu080Iek3vCo3UooG39ZdoAMgk+H scC6LEnR3E87
9lq2HjyT9dO3MwvYBNt+xOvG/ut1pq3ONkOLaM9PVwGiZ8Xm6kl1ioUseIfH L9dIcHgGFxeV
ohFGyDRqEQ91e6VkU2zgXM/6D6Q/JIepEru4sg7nIXTnyKctedH3/xQrSPIe 3yxF5wb4desC
AwEAAaMtMCswGwYDVR0RBBQwEoEQbWluZnJpbkBzaGFycC5mbTAMBgNVHRMB Af8EAjAAMA0G
CSqGSIb3DQEBBAUAA4GBAIEzo1Zm1QIuILSjYVCNlHDFhOes+pxuysUGdhJi UiA4V6KW2bF7
bjroAQ8SNV+PWqskKjpLR4Fo4oovxtKaCoTXotyK1Ljwdm6fc6PDLYsgdm61 4Jb4+TfdI7AP
eDBDLZwy3HZT+Avl5Ms2q21AIfUyccZpXmrXh08mViDmXD0dMIIDPzCCAqig AwIBAgIBDTAN
BgkqhkiG9w0BAQUFADCB0TELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rl cm4gQ2FwZTES
MBAGA1UEBxMJQ2FwZSBUb3duMRowGAYDVQQKExFUaGF3dGUgQ29uc3VsdGlu ZzEoMCYGA1UE
CxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEkMCIGA1UEAxMb VGhhd3RlIFBl
cnNvbmFsIEZyZWVtYWlsIENBMSswKQYJKoZIhvcNAQkBFhxwZXJzb25hbC1m cmVlbWFpbEB0
aGF3dGUuY29tMB4XDTAzMDcxNzAwMDAwMFoXDTEzMDcxNjIzNTk1OVowYjEL MAkGA1UEBhMC
WkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAq BgNVBAMTI1Ro
YXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMIGfMA0GCSqGSIb3 DQEBAQUAA4GN
ADCBiQKBgQDEpjxVc1X7TrnKmVoeaMB1BHCd3+n/ox7svc31W/Iadr1/DDph 8r9RzgHU5VAK
MNcCY1osiRVwjt3J8CuFWqo/cVbLrzwLB+fxH5E2JCoTzyvV84J3PQO+K/67 GD4Hv0CAAmTX
p6a7n2XRxSpUhQ9IBH+nttE8YQRAHmQZcmC3+wIDAQABo4GUMIGRMBIGA1Ud EwEB/wQIMAYB
Af8CAQAwQwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL2NybC50aGF3dGUuY29t L1RoYXd0ZVBl
cnNvbmFsRnJlZW1haWxDQS5jcmwwCwYDVR0PBAQDAgEGMCkGA1UdEQQiMCCk HjAcMRowGAYD
VQQDExFQcml2YXRlTGFiZWwyLTEzODANBgkqhkiG9w0BAQUFAAOBgQBIjNFQ g+oLLswNo2as
Zw9/r6y+whehQ5aUnX9MIbj4Nh+qLZ82L8D0HFAgk3A8/a3hYWLD2ToZfoSx mRsAxRoLgnSe
JVCUYsfbJ3FXJY3dqZw5jowgT2Vfldr394fWxghOrvbqNOUQGls1TXfjViF4 gtwhGTXeJLHT
HUb/XV9lTzGCAzswggM3AgEBMGkwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoT HFRoYXd0ZSBD
b25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25h bCBGcmVlbWFp
bCBJc3N1aW5nIENBAgMMmfAwCQYFKw4DAhoFAKCCAacwGAYJKoZIhvcNAQkD MQsGCSqGSIb3
DQEHATAcBgkqhkiG9w0BCQUxDxcNMDQxMDIwMTM0NjQ5WjAjBgkqhkiG9w0B CQQxFgQUzFe9
OZTCKuE2RARxHIG3fGhV95swUgYJKoZIhvcNAQkPMUUwQzAKBggqhkiG9w0D BzAOBggqhkiG
9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwIC ASgweAYJKwYB
BAGCNxAEMWswaTBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENv bnN1bHRpbmcg
KFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWls IElzc3Vpbmcg
Q0ECAwyZ8DB6BgsqhkiG9w0BCRACCzFroGkwYjELMAkGA1UEBhMCWkExJTAj BgNVBAoTHFRo
YXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQ ZXJzb25hbCBG
cmVlbWFpbCBJc3N1aW5nIENBAgMMmfAwDQYJKoZIhvcNAQEBBQAEggEACwmt qT2e8rQzWHYn
dK3dAvCHZDyFKkAy8AsV8bLZA89YMV/t/vqZ3htexIL3VnEwuPrUV6UMgdKp /SsuUvrrZVtv
22wNa/arld6jm4Yk4WTnOgOBmY41ejBJAjc5akzsbVNsOzo7D+TYjSKtKPtG 4tMtL9zcVZba
GNHPumfhRA5IBGr8fMW+G55ehjdI7BmE3jWIbFUxnk8j0tr+A7EwyZxdJIy2 Y/uAutd8+m7N
S932q3v15rkzPHjkJcZWrG2GASQ/VNkqNOzdGgp/B8FzQKGb/C2Ubr/ar0YT f3548y8IRwq3
4eShQkkI8STsWzHZLvroFxZFicwAyHw/pOjJeQAAAAAAAA==
--------------ms070100040701070201050008--