Debugging a long process

I have a mod_perl Registry script that is taking about 30 seconds to
execute.

I am trying to debug it but when I run the script from the command line
or the debugger it only takes 9 seconds to execute.

I am running Apache/2.2.11 (Ubuntu) mod_perl/2.0.4 Perl/v5.10.0

Is there something obvious I should be looking at, and does anyone have
any tips for debugging something like this?

Thanks!

Tosh

--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: Debugging a long process

On Wed, Jun 8, 2011 at 12:50 PM, Tosh Cooey wrote:
> I have a mod_perl Registry script that is taking about 30 seconds to
> execute.
>
> I am trying to debug it but when I run the script from the command line or
> the debugger it only takes 9 seconds to execute.

Sounds like Apache may be blocking on other processes? Have you tried
Devel::NYTProf with your program on the command line?

>
> I am running Apache/2.2.11 (Ubuntu) mod_perl/2.0.4 Perl/v5.10.0
>
> Is there something obvious I should be looking at, and does anyone have any
> tips for debugging something like this?
>
> Thanks!
>
> Tosh
>
> --
> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
>

Re: Debugging a long process

I haven't tried Devel::NYTProf, but the blocking seems to have been the
keyword here...

root 2661 0.0 1.5 37900 27492 ? Ss Jun08 0:00
/usr/sbin/apache2 -k start
www-data 3875 0.0 1.3 37264 23300 ? S 06:25 0:00
/usr/sbin/apache2 -k start
www-data 3882 3.9 8.7 395156 153644 ? Sl 06:25 6:15
/usr/sbin/apache2 -k start

As you can see I've only got three Apache+mod_perl processes running,
which I suspect is causing the blocking issues since there aren't enough
processes being spawned to handle the incoming requests.

Oddly enough I used to have the problem that this application was
spawning too many processes, now I have the opposite problem.

Anyone have sweet spots for Apache settings running on an EC2 small
instance?

Thanks Fred!

Tosh


On 6/8/11 10:15 PM, Fred Moyer wrote:
> On Wed, Jun 8, 2011 at 12:50 PM, Tosh Cooey wrote:
>> I have a mod_perl Registry script that is taking about 30 seconds to
>> execute.
>>
>> I am trying to debug it but when I run the script from the command line or
>> the debugger it only takes 9 seconds to execute.
>
> Sounds like Apache may be blocking on other processes? Have you tried
> Devel::NYTProf with your program on the command line?
>
>>
>> I am running Apache/2.2.11 (Ubuntu) mod_perl/2.0.4 Perl/v5.10.0
>>
>> Is there something obvious I should be looking at, and does anyone have any
>> tips for debugging something like this?
>>
>> Thanks!
>>
>> Tosh
>>
>> --
>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
>>
>

--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: Debugging a long process

On 9 Jun 2011, at 17:13, Tosh Cooey wrote:

> I haven't tried Devel::NYTProf, but the blocking seems to have been =
the keyword here...
>=20
> root 2661 0.0 1.5 37900 27492 ? Ss Jun08 0:00 =
/usr/sbin/apache2 -k start
> www-data 3875 0.0 1.3 37264 23300 ? S 06:25 0:00 =
/usr/sbin/apache2 -k start
> www-data 3882 3.9 8.7 395156 153644 ? Sl 06:25 6:15 =
/usr/sbin/apache2 -k start

Very much looks like you don't have a startup.pl that loads the world in =
the parent=20
process. That will cause a masive inhale on the first hit.

>=20
> Anyone have sweet spots for Apache settings running on an EC2 small =
instance?
> =20

Whatever works for you after measurement.

Read the mod_perl tuning guide and settle on a strategy of =
MaxRequestsPerClient
or a fixed Apache2::SizeLimit.

Also, are you running stock apache/perl/mod_perl? Seriously consider =
spending
time making your own build. Even recompiling the .srpms with gentoo-safe =
flags
for that $arch could give a healthy boost.

Or pay someone *cough* *cough* to do it for you :)

Re: Re: Debugging a long process

Hi Fred, I like the idea of "Apache may be blocking on other processes"
but the Devel::NYTProf docs don't seem to indicate if it would be
helpful in tracking down the reason for a 30 second execution under
mod_perl versus a 9 second execution from the shell. Also there seems
to be some issues WRT Devel::NYTProf and virtual machines, which I
believe my EC2 instance is one of.

Is there anywhere to point me to find out more about the blocking issue
and how to resolve it?

Thanks!

Tosh


On 7/22/64 8:59 PM, Fred Moyer wrote:
> On Wed, Jun 8, 2011 at 12:50 PM, Tosh Cooey wrote:
>> I have a mod_perl Registry script that is taking about 30 seconds to
>> execute.
>>
>> I am trying to debug it but when I run the script from the command line or
>> the debugger it only takes 9 seconds to execute.
>
> Sounds like Apache may be blocking on other processes? Have you tried
> Devel::NYTProf with your program on the command line?
>
>>
>> I am running Apache/2.2.11 (Ubuntu) mod_perl/2.0.4 Perl/v5.10.0
>>
>> Is there something obvious I should be looking at, and does anyone have any
>> tips for debugging something like this?
>>
>> Thanks!
>>
>> Tosh
>>
>> --
>> McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
>>
>

--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: Debugging a long process

Hey Tosh

From my experience, I would guess you may seeing interactions between the
major components of your system as load grows.
If my hunch is true, then following a single request may not show the
issue.
You may need a way to load your system to the point where it shows the
issue.
If all these guesses are true, NYTProf, while an excellent tool, didn't
really help me and may not be much help to you.

You mention you may be running in an EC2 instance, which adds another
layer at which your code may be getting suspended.

When I faced a similar problem, I resolved it by:

- create a test environment that I could push to show the problem
- instrument major blocks of the code with markers and tv_interval() calls
so I could see where time was being spent.
- write a row of timing data for each request
- post process the timing data spread over, in my case, many many
individual requests so I could see how response time changed as load
increased

This took me a few days to get worked out, but once I had it in place, I
was able to tune my system more effectively.
I could 'see' where the bottlenecks were occurring as load increased.
In my case, it pointed to a single component in my stack which was the
bottleneck.
So I fixed that one, then fixed the next hot spot ... you get the picture.

There are a lot of assumptions here that may or may not actually be true
in your case...
HTH,

-- jeff


On 6/15/11 1:29 PM, "Tosh Cooey" wrote:

>Hi Fred, I like the idea of "Apache may be blocking on other processes"
>but the Devel::NYTProf docs don't seem to indicate if it would be
>helpful in tracking down the reason for a 30 second execution under
>mod_perl versus a 9 second execution from the shell. Also there seems
>to be some issues WRT Devel::NYTProf and virtual machines, which I
>believe my EC2 instance is one of.
>
>Is there anywhere to point me to find out more about the blocking issue
>and how to resolve it?

Re: Re: Debugging a long process

On Wed, Jun 15, 2011 at 1:29 PM, Tosh Cooey wrote:
> Hi Fred, I like the idea of "Apache may be blocking on other processes" b=
ut
> the Devel::NYTProf docs don't seem to indicate if it would be helpful in
> tracking down the reason for a 30 second execution under mod_perl versus =
a 9
> second execution from the shell. =A0Also there seems to be some issues WR=
T
> Devel::NYTProf and virtual machines, which I believe my EC2 instance is o=
ne
> of.

Have you tried running your program under Devel::NYTProf in your
development environment and using that to identify bottlenecks?

Re: Debugging a long process

Hi Jeff, this is very sensible and will likely have to be what gets done.

To respond to Fred, my DEV environment is also a small EC2 instance
since I want to mirror my production environment. This line of thought
prodded me to check my environments and it seems that the PROD Apache is
running the worker MPM while DEV is running prefork, and boy is there a
performance difference.

I am waiting to hear back from my server admin before I kill him, but it
seems that maybe there's some issues with my Ubuntu and the worker MPM
on a small EC2 instance.

Thank-you both for your insights!

Tosh


On 6/15/11 11:00 PM, McCarrell, Jeff wrote:
> Hey Tosh
>
>> From my experience, I would guess you may seeing interactions between the
> major components of your system as load grows.
> If my hunch is true, then following a single request may not show the
> issue.
> You may need a way to load your system to the point where it shows the
> issue.
> If all these guesses are true, NYTProf, while an excellent tool, didn't
> really help me and may not be much help to you.
>
> You mention you may be running in an EC2 instance, which adds another
> layer at which your code may be getting suspended.
>
> When I faced a similar problem, I resolved it by:
>
> - create a test environment that I could push to show the problem
> - instrument major blocks of the code with markers and tv_interval() calls
> so I could see where time was being spent.
> - write a row of timing data for each request
> - post process the timing data spread over, in my case, many many
> individual requests so I could see how response time changed as load
> increased
>
> This took me a few days to get worked out, but once I had it in place, I
> was able to tune my system more effectively.
> I could 'see' where the bottlenecks were occurring as load increased.
> In my case, it pointed to a single component in my stack which was the
> bottleneck.
> So I fixed that one, then fixed the next hot spot ... you get the picture.
>
> There are a lot of assumptions here that may or may not actually be true
> in your case...
> HTH,
>
> -- jeff
>
>
> On 6/15/11 1:29 PM, "Tosh Cooey" wrote:
>
>> Hi Fred, I like the idea of "Apache may be blocking on other processes"
>> but the Devel::NYTProf docs don't seem to indicate if it would be
>> helpful in tracking down the reason for a 30 second execution under
>> mod_perl versus a 9 second execution from the shell. Also there seems
>> to be some issues WRT Devel::NYTProf and virtual machines, which I
>> believe my EC2 instance is one of.
>>
>> Is there anywhere to point me to find out more about the blocking issue
>> and how to resolve it?
>
>

--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/

Re: Debugging a long process

On 15 Jun 2011, at 22:53, Tosh Cooey wrote:
>=20
> I am waiting to hear back from my server admin before I kill him, but =
it seems that maybe there's some issues with my Ubuntu and the worker =
MPM on a small EC2 instance.

I've been doing this stuff since web 1.0 and have always found prefork =
to=20
be sensible and deterministic.

I'd go with the defenestration option.