mod_ldap cache and strace doesn"t tell the same story

Hi all,

I'm facing a strange issue.
I have a major performance penalty on some apache related operations
(svn checkout that used to take 5mn but that suddenly takes more than
7 hours).

I'm running a standard RHEL5.3 with apache 2.2.3 (fyi the server runs
mod_php, mod_auth_mysql and some other modules too).

Thanks to strace, I identified that the httpd process that is "very
long" (the httpd process that discuss with svn client that takes hour
to complete).
The process makes a lot of operation on a file descriptor attached to
my ldap server and is regulary blocking with following syscall:
poll([{fd=3, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, -1

fd3 is a socket (/proc//fd/3) and the inode of this socket is
used to communicate with the ldap server (shown by netstat)

This is where I no longer understand what happens:
- I'm using mod_ldap so there should be a cache of all ldap info.
- ldap-status confirm that the credentials used for the svn operation
are in cache.
- The cache is far from full (2%)
- The hit ratio is close to 100%

=> Why there is an activity between my httpd process and the ldap server ?

Any help appreciate,
Cheers,
Manuel

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: mod_ldap cache and strace doesn"t tell the same story

On Tue, Mar 16, 2010 at 11:06 AM, Manuel Vacelet
wrote:
> Hi all,
>
> I'm facing a strange issue.
> I have a major performance penalty on some apache related operations
> (svn checkout that used to take 5mn but that suddenly takes more than
> 7 hours).
>
> I'm running a standard RHEL5.3 with apache 2.2.3 (fyi the server runs
> mod_php, mod_auth_mysql and some other modules too).
>
> Thanks to strace, I identified that the httpd process that is "very
> long" (the httpd process that discuss with svn client that takes hour
> to complete).
> The process makes a lot of operation on a file descriptor attached to
> my ldap server and is regulary blocking with following syscall:
> poll([{fd=3, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, -1
>
> fd3 is a socket (/proc//fd/3) and the inode of this socket is
> used to communicate with the ldap server (shown by netstat)
>
> This is where I no longer understand what happens:
> - I'm using mod_ldap so there should be a cache of all ldap info.
> - ldap-status confirm that the credentials used for the svn operation
> are in cache.
> - The cache is far from full (2%)
> - The hit ratio is close to 100%
>
> => Why there is an activity between my httpd process and the ldap server ?

I think I have an answer to this question:
In some cases, the apache process (I'm running prefork) doesn't use
mod_ldap cache.

I made a test:
In parallel of a very slow checkout, I ran another one. The second
checkout was running as normal rate.
A couple of strace on the server later, it appears that, the apache
process that server the second checkout never talk to the ldap server

So the new question is:
Why 2 apache processes forked from the same root could behave so differently ?

Manuel

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: mod_ldap cache and strace doesn"t tell the same story

On Tue, Mar 16, 2010 at 2:18 PM, Manuel Vacelet
wrote:
> On Tue, Mar 16, 2010 at 11:06 AM, Manuel Vacelet
> wrote:
>> Hi all,
>>
>> I'm facing a strange issue.
>> I have a major performance penalty on some apache related operations
>> (svn checkout that used to take 5mn but that suddenly takes more than
>> 7 hours).
>>
>> I'm running a standard RHEL5.3 with apache 2.2.3 (fyi the server runs
>> mod_php, mod_auth_mysql and some other modules too).
>>
>> Thanks to strace, I identified that the httpd process that is "very
>> long" (the httpd process that discuss with svn client that takes hour
>> to complete).
>> The process makes a lot of operation on a file descriptor attached to
>> my ldap server and is regulary blocking with following syscall:
>> poll([{fd=3, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, -1
>>
>> fd3 is a socket (/proc//fd/3) and the inode of this socket is
>> used to communicate with the ldap server (shown by netstat)
>>
>> This is where I no longer understand what happens:
>> - I'm using mod_ldap so there should be a cache of all ldap info.
>> - ldap-status confirm that the credentials used for the svn operation
>> are in cache.
>> - The cache is far from full (2%)
>> - The hit ratio is close to 100%
>>
>> => Why there is an activity between my httpd process and the ldap server ?
>
> I think I have an answer to this question:
> In some cases, the apache process (I'm running prefork) doesn't use
> mod_ldap cache.
>
> I made a test:
> In parallel of a very slow checkout, I ran another one. The second
> checkout was running as normal rate.
> A couple of strace on the server later, it appears that, the apache
> process that server the second checkout never talk to the ldap server
>
> So the new question is:
> Why 2 apache processes forked from the same root could behave so differently ?

Does anybody have an explanation to this whole thing ?

Why "sometimes" mod_ldap never cache my credentials ?
Or in other words: "what would make mod_ldap not to cache users credentials" ?

Manuel

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: mod_ldap cache and strace doesn"t tell the same story

I continue my investigations...

The mod_ldap cache failure seems related to regular graceful restart
(~30mn) the application send to apache to take into account conf
update.

Does it make sense ?
How graceful restart could impact caching of mod_ldap?

Manuel

On Wed, Mar 17, 2010 at 4:16 PM, Manuel Vacelet
wrote:
> On Tue, Mar 16, 2010 at 2:18 PM, Manuel Vacelet
> wrote:
>> On Tue, Mar 16, 2010 at 11:06 AM, Manuel Vacelet
>> wrote:
>>> Hi all,
>>>
>>> I'm facing a strange issue.
>>> I have a major performance penalty on some apache related operations
>>> (svn checkout that used to take 5mn but that suddenly takes more than
>>> 7 hours).
>>>
>>> I'm running a standard RHEL5.3 with apache 2.2.3 (fyi the server runs
>>> mod_php, mod_auth_mysql and some other modules too).
>>>
>>> Thanks to strace, I identified that the httpd process that is "very
>>> long" (the httpd process that discuss with svn client that takes hour
>>> to complete).
>>> The process makes a lot of operation on a file descriptor attached to
>>> my ldap server and is regulary blocking with following syscall:
>>> poll([{fd=3, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, -1
>>>
>>> fd3 is a socket (/proc//fd/3) and the inode of this socket is
>>> used to communicate with the ldap server (shown by netstat)
>>>
>>> This is where I no longer understand what happens:
>>> - I'm using mod_ldap so there should be a cache of all ldap info.
>>> - ldap-status confirm that the credentials used for the svn operation
>>> are in cache.
>>> - The cache is far from full (2%)
>>> - The hit ratio is close to 100%
>>>
>>> => Why there is an activity between my httpd process and the ldap server ?
>>
>> I think I have an answer to this question:
>> In some cases, the apache process (I'm running prefork) doesn't use
>> mod_ldap cache.
>>
>> I made a test:
>> In parallel of a very slow checkout, I ran another one. The second
>> checkout was running as normal rate.
>> A couple of strace on the server later, it appears that, the apache
>> process that server the second checkout never talk to the ldap server
>>
>> So the new question is:
>> Why 2 apache processes forked from the same root could behave so differently ?
>
> Does anybody have an explanation to this whole thing ?
>
> Why "sometimes" mod_ldap never cache my credentials ?
> Or in other words: "what would make mod_ldap not to cache users credentials" ?
>
> Manuel
>

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: mod_ldap cache and strace doesn"t tell the same story

Hi,

Just to keep people informed (and search engines). It seems to be a
httpd bug on usage of LDAP cache shared memory and graceful restart:
https://issues.apache.org/bugzilla/show_bug.cgi?id=48958

Manuel

On Fri, Mar 19, 2010 at 9:34 AM, Manuel Vacelet
wrote:
> I continue my investigations...
>
> The mod_ldap cache failure seems related to regular graceful restart
> (~30mn) the application send to apache to take into account conf
> update.
>
> Does it make sense ?
> How graceful restart could impact caching of mod_ldap?
>
> Manuel
>
> On Wed, Mar 17, 2010 at 4:16 PM, Manuel Vacelet
> wrote:
>> On Tue, Mar 16, 2010 at 2:18 PM, Manuel Vacelet
>> wrote:
>>> On Tue, Mar 16, 2010 at 11:06 AM, Manuel Vacelet
>>> wrote:
>>>> Hi all,
>>>>
>>>> I'm facing a strange issue.
>>>> I have a major performance penalty on some apache related operations
>>>> (svn checkout that used to take 5mn but that suddenly takes more than
>>>> 7 hours).
>>>>
>>>> I'm running a standard RHEL5.3 with apache 2.2.3 (fyi the server runs
>>>> mod_php, mod_auth_mysql and some other modules too).
>>>>
>>>> Thanks to strace, I identified that the httpd process that is "very
>>>> long" (the httpd process that discuss with svn client that takes hour
>>>> to complete).
>>>> The process makes a lot of operation on a file descriptor attached to
>>>> my ldap server and is regulary blocking with following syscall:
>>>> poll([{fd=3, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, -1
>>>>
>>>> fd3 is a socket (/proc//fd/3) and the inode of this socket is
>>>> used to communicate with the ldap server (shown by netstat)
>>>>
>>>> This is where I no longer understand what happens:
>>>> - I'm using mod_ldap so there should be a cache of all ldap info.
>>>> - ldap-status confirm that the credentials used for the svn operation
>>>> are in cache.
>>>> - The cache is far from full (2%)
>>>> - The hit ratio is close to 100%
>>>>
>>>> => Why there is an activity between my httpd process and the ldap server ?
>>>
>>> I think I have an answer to this question:
>>> In some cases, the apache process (I'm running prefork) doesn't use
>>> mod_ldap cache.
>>>
>>> I made a test:
>>> In parallel of a very slow checkout, I ran another one. The second
>>> checkout was running as normal rate.
>>> A couple of strace on the server later, it appears that, the apache
>>> process that server the second checkout never talk to the ldap server
>>>
>>> So the new question is:
>>> Why 2 apache processes forked from the same root could behave so differently ?
>>
>> Does anybody have an explanation to this whole thing ?
>>
>> Why "sometimes" mod_ldap never cache my credentials ?
>> Or in other words: "what would make mod_ldap not to cache users credentials" ?
>>
>> Manuel
>>
>

------------------------------------------------------------ ---------
The official User-To-User support forum of the Apache HTTP Server Project.
See for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org