High CPU utilization on RHEL5.6/CentOS5.6

I have a MJPEG streaming system which uses mod_perl in the web
interface to supply the final stream to the client.

I am seeing high cpu utilization under RHEL5.6 which I don't see on
FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on
different hardware, but not that different. And on EL5 the httpd
session is way higher (cpu wise) than the supplying proxy. On FC14 on
my test rig I see (for a 25fps stream) 15-18% CPU for the proxy which
can handle multiplexing etc and 5.5-8% for the mod_perl httpd session.

I think this is an issue with that platform, but I haven't seen
any/many reports. Does anyone have a sec to look at the source? I
don't think I have implemented this is a horrible inefficient way but
who knows. It is true, I could have byte markers and then know the
size of the JPEG I am expecting rather than just splitting on the
boundary field, but in tests (and production) it seemed to work ok.

http://svn.codaxus.com/flexTPS/2.x.x/trunk/portal/2.4.x/site /perl/nph-mjpeg_stream.pl

Any comments or critiques would be appreciated,
Christopher

------------------------------------------------------------ ----------

On FC14:
Requires: httpd >= 2.2.17-1
Requires: mod_perl >= 2.0.4-11
Requires: mod_ssl >= 1:2.2.17-1
Requires: perl >= 5.12.3-141
Requires: perl-CGI >= 3.51-1
Requires: perl-Digest-SHA1 >= 2.12-4
Requires: perl-XML-Filter-BufferText >= 1.01-9
Requires: perl-XML-Simple >= 2.18-7
Requires: perl-XML-Validator-Schema >= 1.10-5
Requires: perl-Tree-DAG_Node >= 1.06-8
Requires: perl-LDAP >= 0.40-2
Requires: perl-Crypt-SSLeay >= 0.58-1
Requires: ffmpeg >= 0.6-4

On EL5:
Requires: httpd >= 2.2.3-45
Requires: mod_perl >= 2.0.4-6
Requires: mod_ssl >= 2.2.3-45
Requires: perl >= 5.8.8-32
Requires: perl-Digest-SHA1 >= 2.11-1
Requires: perl-XML-Filter-BufferText >= 1.01
Requires: perl-XML-Simple >= 2.14-4
Requires: perl-XML-Validator-Schema >= 1.10-1
Requires: perl-Tree-DAG_Node >= 1.06
Requires: perl-LDAP >= 0.33-3
Requires: perl-Crypt-SSLeay >= 0.51-11
Requires: ffmpeg >= 0.6.1-1

Re: High CPU utilization on RHEL5.6/CentOS5.6

This may not be related, but when I was working with Centos 5 in a
Vmware environment, I ran into an issue of high cpu since the default
clock rate with Linux 2.6 is 1000Hz. I changed to a 100Hz clock rate
and got much lower cpu usage.

I'm not sure if that is your issue, perhaps FC14 was using less than a
1000Hz clock rate. You might try a tickless kernel, or dig further
into what clock rate your kernel is using.

I don't know where the authoritative source for this information is,
but here's what I found with a search:

http://yate.null.ro/pmwiki/index.php?n=Main.YateAndVMWare

On Mon, Jul 25, 2011 at 3:43 PM, Christopher Stanton
wrote:
> I have a MJPEG streaming system which uses mod_perl in the web
> interface to supply the final stream to the client.
>
> I am seeing high cpu utilization under RHEL5.6 which I don't see on
> FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on
> different hardware, but not that different. And on EL5 the httpd
> session is way higher (cpu wise) than the supplying proxy. On FC14 on
> my test rig I see (for a 25fps stream) 15-18% CPU for the proxy which
> can handle multiplexing etc and 5.5-8% for the mod_perl httpd session.
>
> I think this is an issue with that platform, but I haven't seen
> any/many reports. Does anyone have a sec to look at the source? I
> don't think I have implemented this is a horrible inefficient way but
> who knows. It is true, I could have byte markers and then know the
> size of the JPEG I am expecting rather than just splitting on the
> boundary field, but in tests (and production) it seemed to work ok.
>
> http://svn.codaxus.com/flexTPS/2.x.x/trunk/portal/2.4.x/site /perl/nph-mjpeg_stream.pl
>
> Any comments or critiques would be appreciated,
> Christopher
>
> ------------------------------------------------------------ ----------
>
> On FC14:
> Requires: httpd >= 2.2.17-1
> Requires: mod_perl >= 2.0.4-11
> Requires: mod_ssl >= 1:2.2.17-1
> Requires: perl >= 5.12.3-141
> Requires: perl-CGI >= 3.51-1
> Requires: perl-Digest-SHA1 >= 2.12-4
> Requires: perl-XML-Filter-BufferText >= 1.01-9
> Requires: perl-XML-Simple >= 2.18-7
> Requires: perl-XML-Validator-Schema >= 1.10-5
> Requires: perl-Tree-DAG_Node >= 1.06-8
> Requires: perl-LDAP >= 0.40-2
> Requires: perl-Crypt-SSLeay >= 0.58-1
> Requires: ffmpeg >= 0.6-4
>
> On EL5:
> Requires: httpd >= 2.2.3-45
> Requires: mod_perl >= 2.0.4-6
> Requires: mod_ssl >= 2.2.3-45
> Requires: perl >= 5.8.8-32
> Requires: perl-Digest-SHA1 >= 2.11-1
> Requires: perl-XML-Filter-BufferText >= 1.01
> Requires: perl-XML-Simple >= 2.14-4
> Requires: perl-XML-Validator-Schema >= 1.10-1
> Requires: perl-Tree-DAG_Node >= 1.06
> Requires: perl-LDAP >= 0.33-3
> Requires: perl-Crypt-SSLeay >= 0.51-11
> Requires: ffmpeg >= 0.6.1-1
>

Re: High CPU utilization on RHEL5.6/CentOS5.6

Neither install is running within VMWare.

On Mon, Jul 25, 2011 at 5:55 PM, Fred Moyer wrote:
> This may not be related, but when I was working with Centos 5 in a
> Vmware environment, I ran into an issue of high cpu since the default
> clock rate with Linux 2.6 is 1000Hz. =A0I changed to a 100Hz clock rate
> and got much lower cpu usage.
>
> I'm not sure if that is your issue, perhaps FC14 was using less than a
> 1000Hz clock rate. =A0You might try a tickless kernel, or dig further
> into what clock rate your kernel is using.
>
> I don't know where the authoritative source for this information is,
> but here's what I found with a search:
>
> http://yate.null.ro/pmwiki/index.php?n=3DMain.YateAndVMWare
>
> On Mon, Jul 25, 2011 at 3:43 PM, Christopher Stanton
> wrote:
>> I have a MJPEG streaming system which uses mod_perl in the web
>> interface to supply the final stream to the client.
>>
>> I am seeing high cpu utilization under RHEL5.6 which I don't see on
>> FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on
>> different hardware, but not that different. And on EL5 the httpd
>> session is way higher (cpu wise) than the supplying proxy. On FC14 on
>> my test rig I see (for a 25fps stream) 15-18% CPU for the proxy which
>> can handle multiplexing etc and 5.5-8% for the mod_perl httpd session.
>>
>> I think this is an issue with that platform, but I haven't seen
>> any/many reports. Does anyone have a sec to look at the source? I
>> don't think I have implemented this is a horrible inefficient way but
>> who knows. It is true, I could have byte markers and then know the
>> size of the JPEG I am expecting rather than just splitting on the
>> boundary field, but in tests (and production) it seemed to work ok.
>>
>> http://svn.codaxus.com/flexTPS/2.x.x/trunk/portal/2.4.x/site /perl/nph-mj=
peg_stream.pl
>>
>> Any comments or critiques would be appreciated,
>> Christopher
>>
>> ------------------------------------------------------------ ----------
>>
>> On FC14:
>> Requires: httpd >=3D 2.2.17-1
>> Requires: mod_perl >=3D 2.0.4-11
>> Requires: mod_ssl >=3D 1:2.2.17-1
>> Requires: perl >=3D 5.12.3-141
>> Requires: perl-CGI >=3D 3.51-1
>> Requires: perl-Digest-SHA1 >=3D 2.12-4
>> Requires: perl-XML-Filter-BufferText >=3D 1.01-9
>> Requires: perl-XML-Simple >=3D 2.18-7
>> Requires: perl-XML-Validator-Schema >=3D 1.10-5
>> Requires: perl-Tree-DAG_Node >=3D 1.06-8
>> Requires: perl-LDAP >=3D 0.40-2
>> Requires: perl-Crypt-SSLeay >=3D 0.58-1
>> Requires: ffmpeg >=3D 0.6-4
>>
>> On EL5:
>> Requires: httpd >=3D 2.2.3-45
>> Requires: mod_perl >=3D 2.0.4-6
>> Requires: mod_ssl >=3D 2.2.3-45
>> Requires: perl >=3D 5.8.8-32
>> Requires: perl-Digest-SHA1 >=3D 2.11-1
>> Requires: perl-XML-Filter-BufferText >=3D 1.01
>> Requires: perl-XML-Simple >=3D 2.14-4
>> Requires: perl-XML-Validator-Schema >=3D 1.10-1
>> Requires: perl-Tree-DAG_Node >=3D 1.06
>> Requires: perl-LDAP >=3D 0.33-3
>> Requires: perl-Crypt-SSLeay >=3D 0.51-11
>> Requires: ffmpeg >=3D 0.6.1-1
>>
>

Re: High CPU utilization on RHEL5.6/CentOS5.6

On Mon, Jul 25, 2011 at 4:02 PM, Christopher Stanton
wrote:
> Neither install is running within VMWare.

Might be worthwhile trying a different rate though, or going tickless.
Perl 5.8.8 isn't that much (if at all) slower than 5.12. Looks like
the FC14 kernel is tickless:

http://docs.fedoraproject.org/en-US/Fedora/14/pdf/Power_Mana gement_Guide/Fe=
dora-14-Power_Management_Guide-en-US.pdf


> On Mon, Jul 25, 2011 at 5:55 PM, Fred Moyer wrot=
e:
>> This may not be related, but when I was working with Centos 5 in a
>> Vmware environment, I ran into an issue of high cpu since the default
>> clock rate with Linux 2.6 is 1000Hz. =A0I changed to a 100Hz clock rate
>> and got much lower cpu usage.
>>
>> I'm not sure if that is your issue, perhaps FC14 was using less than a
>> 1000Hz clock rate. =A0You might try a tickless kernel, or dig further
>> into what clock rate your kernel is using.
>>
>> I don't know where the authoritative source for this information is,
>> but here's what I found with a search:
>>
>> http://yate.null.ro/pmwiki/index.php?n=3DMain.YateAndVMWare
>>
>> On Mon, Jul 25, 2011 at 3:43 PM, Christopher Stanton
>> wrote:
>>> I have a MJPEG streaming system which uses mod_perl in the web
>>> interface to supply the final stream to the client.
>>>
>>> I am seeing high cpu utilization under RHEL5.6 which I don't see on
>>> FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on
>>> different hardware, but not that different. And on EL5 the httpd
>>> session is way higher (cpu wise) than the supplying proxy. On FC14 on
>>> my test rig I see (for a 25fps stream) 15-18% CPU for the proxy which
>>> can handle multiplexing etc and 5.5-8% for the mod_perl httpd session.
>>>
>>> I think this is an issue with that platform, but I haven't seen
>>> any/many reports. Does anyone have a sec to look at the source? I
>>> don't think I have implemented this is a horrible inefficient way but
>>> who knows. It is true, I could have byte markers and then know the
>>> size of the JPEG I am expecting rather than just splitting on the
>>> boundary field, but in tests (and production) it seemed to work ok.
>>>
>>> http://svn.codaxus.com/flexTPS/2.x.x/trunk/portal/2.4.x/site /perl/nph-m=
jpeg_stream.pl
>>>
>>> Any comments or critiques would be appreciated,
>>> Christopher
>>>
>>> ------------------------------------------------------------ ----------
>>>
>>> On FC14:
>>> Requires: httpd >=3D 2.2.17-1
>>> Requires: mod_perl >=3D 2.0.4-11
>>> Requires: mod_ssl >=3D 1:2.2.17-1
>>> Requires: perl >=3D 5.12.3-141
>>> Requires: perl-CGI >=3D 3.51-1
>>> Requires: perl-Digest-SHA1 >=3D 2.12-4
>>> Requires: perl-XML-Filter-BufferText >=3D 1.01-9
>>> Requires: perl-XML-Simple >=3D 2.18-7
>>> Requires: perl-XML-Validator-Schema >=3D 1.10-5
>>> Requires: perl-Tree-DAG_Node >=3D 1.06-8
>>> Requires: perl-LDAP >=3D 0.40-2
>>> Requires: perl-Crypt-SSLeay >=3D 0.58-1
>>> Requires: ffmpeg >=3D 0.6-4
>>>
>>> On EL5:
>>> Requires: httpd >=3D 2.2.3-45
>>> Requires: mod_perl >=3D 2.0.4-6
>>> Requires: mod_ssl >=3D 2.2.3-45
>>> Requires: perl >=3D 5.8.8-32
>>> Requires: perl-Digest-SHA1 >=3D 2.11-1
>>> Requires: perl-XML-Filter-BufferText >=3D 1.01
>>> Requires: perl-XML-Simple >=3D 2.14-4
>>> Requires: perl-XML-Validator-Schema >=3D 1.10-1
>>> Requires: perl-Tree-DAG_Node >=3D 1.06
>>> Requires: perl-LDAP >=3D 0.33-3
>>> Requires: perl-Crypt-SSLeay >=3D 0.51-11
>>> Requires: ffmpeg >=3D 0.6.1-1
>>>
>>
>

Re: High CPU utilization on RHEL5.6/CentOS5.6

Before I start playing mix-and-match with kernels, I would prefer some
feedback on mod_perl and the script's code.

It doesn't sleep in the main loop, but it should block on
my @ready =3D $select->can_read(1);
and not block on
if ( $sock->read(my $data, POSIX::BUFSIZ*10) ){

since the socket is set with Blocking =3D> 0.

A JPEG (in a 25fps stream) arrives every 40,000 microseconds.

Christopher


On Mon, Jul 25, 2011 at 6:43 PM, Fred Moyer wrote:
> On Mon, Jul 25, 2011 at 4:02 PM, Christopher Stanton
> wrote:
>> Neither install is running within VMWare.
>
> Might be worthwhile trying a different rate though, or going tickless.
> =A0Perl 5.8.8 isn't that much (if at all) slower than 5.12. =A0Looks like
> the FC14 kernel is tickless:
>
> http://docs.fedoraproject.org/en-US/Fedora/14/pdf/Power_Mana gement_Guide/=
Fedora-14-Power_Management_Guide-en-US.pdf
>
>
>> On Mon, Jul 25, 2011 at 5:55 PM, Fred Moyer wro=
te:
>>> This may not be related, but when I was working with Centos 5 in a
>>> Vmware environment, I ran into an issue of high cpu since the default
>>> clock rate with Linux 2.6 is 1000Hz. =A0I changed to a 100Hz clock rate
>>> and got much lower cpu usage.
>>>
>>> I'm not sure if that is your issue, perhaps FC14 was using less than a
>>> 1000Hz clock rate. =A0You might try a tickless kernel, or dig further
>>> into what clock rate your kernel is using.
>>>
>>> I don't know where the authoritative source for this information is,
>>> but here's what I found with a search:
>>>
>>> http://yate.null.ro/pmwiki/index.php?n=3DMain.YateAndVMWare
>>>
>>> On Mon, Jul 25, 2011 at 3:43 PM, Christopher Stanton
>>> wrote:
>>>> I have a MJPEG streaming system which uses mod_perl in the web
>>>> interface to supply the final stream to the client.
>>>>
>>>> I am seeing high cpu utilization under RHEL5.6 which I don't see on
>>>> FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on
>>>> different hardware, but not that different. And on EL5 the httpd
>>>> session is way higher (cpu wise) than the supplying proxy. On FC14 on
>>>> my test rig I see (for a 25fps stream) 15-18% CPU for the proxy which
>>>> can handle multiplexing etc and 5.5-8% for the mod_perl httpd session.
>>>>
>>>> I think this is an issue with that platform, but I haven't seen
>>>> any/many reports. Does anyone have a sec to look at the source? I
>>>> don't think I have implemented this is a horrible inefficient way but
>>>> who knows. It is true, I could have byte markers and then know the
>>>> size of the JPEG I am expecting rather than just splitting on the
>>>> boundary field, but in tests (and production) it seemed to work ok.
>>>>
>>>> http://svn.codaxus.com/flexTPS/2.x.x/trunk/portal/2.4.x/site /perl/nph-=
mjpeg_stream.pl
>>>>
>>>> Any comments or critiques would be appreciated,
>>>> Christopher
>>>>
>>>> ------------------------------------------------------------ ----------
>>>>
>>>> On FC14:
>>>> Requires: httpd >=3D 2.2.17-1
>>>> Requires: mod_perl >=3D 2.0.4-11
>>>> Requires: mod_ssl >=3D 1:2.2.17-1
>>>> Requires: perl >=3D 5.12.3-141
>>>> Requires: perl-CGI >=3D 3.51-1
>>>> Requires: perl-Digest-SHA1 >=3D 2.12-4
>>>> Requires: perl-XML-Filter-BufferText >=3D 1.01-9
>>>> Requires: perl-XML-Simple >=3D 2.18-7
>>>> Requires: perl-XML-Validator-Schema >=3D 1.10-5
>>>> Requires: perl-Tree-DAG_Node >=3D 1.06-8
>>>> Requires: perl-LDAP >=3D 0.40-2
>>>> Requires: perl-Crypt-SSLeay >=3D 0.58-1
>>>> Requires: ffmpeg >=3D 0.6-4
>>>>
>>>> On EL5:
>>>> Requires: httpd >=3D 2.2.3-45
>>>> Requires: mod_perl >=3D 2.0.4-6
>>>> Requires: mod_ssl >=3D 2.2.3-45
>>>> Requires: perl >=3D 5.8.8-32
>>>> Requires: perl-Digest-SHA1 >=3D 2.11-1
>>>> Requires: perl-XML-Filter-BufferText >=3D 1.01
>>>> Requires: perl-XML-Simple >=3D 2.14-4
>>>> Requires: perl-XML-Validator-Schema >=3D 1.10-1
>>>> Requires: perl-Tree-DAG_Node >=3D 1.06
>>>> Requires: perl-LDAP >=3D 0.33-3
>>>> Requires: perl-Crypt-SSLeay >=3D 0.51-11
>>>> Requires: ffmpeg >=3D 0.6.1-1
>>>>
>>>
>>
>

Re: High CPU utilization on RHEL5.6/CentOS5.6

On Mon, Jul 25, 2011 at 6:43 PM, Christopher Stanton
wrote:
> I am seeing high cpu utilization under RHEL5.6 which I don't see on
> FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on
> different hardware, but not that different.

Well, you've changed hardware and lots of software versions, so there
are many things it could be. The only one that seems unlikely is your
code, since that didn't change.

I would suspect the major downgrade in Perl as the culprit. You went
from 5.12 to 5.8. To test this, I suggest you install 5.12 on your
new system and compile mod_perl against it. If you don't want to
touch the system mod_perl and apache you can install your own in a
different location. I use perlbrew to try out new Perl versions
locally.

- Perrin

Re: High CPU utilization on RHEL5.6/CentOS5.6

I will see if I can get Perl 5.12 and an associated mod_perl up and
running on the problem machine and see if the CPU utilization goes
down.

Long long ago in a install far far away I did not run into this issue
on FC4 which has very similar packages other than apache (2.0 rather
than 2.2):
Requires: httpd >=3D 2.0.54-0; perl >=3D 5.8.6-0; perl-Digest-SHA1 >=3D
2.10-0; mod_perl >=3D 2.0.1-0;

On Tue, Jul 26, 2011 at 12:43 PM, Perrin Harkins wrote:
> On Mon, Jul 25, 2011 at 6:43 PM, Christopher Stanton
> wrote:
>> I am seeing high cpu utilization under RHEL5.6 which I don't see on
>> FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on
>> different hardware, but not that different.
>
> Well, you've changed hardware and lots of software versions, so there
> are many things it could be. =A0The only one that seems unlikely is your
> code, since that didn't change.
>
> I would suspect the major downgrade in Perl as the culprit. =A0You went
> from 5.12 to 5.8. =A0To test this, I suggest you install 5.12 on your
> new system and compile mod_perl against it. =A0If you don't want to
> touch the system mod_perl and apache you can install your own in a
> different location. =A0I use perlbrew to try out new Perl versions
> locally.
>
> - Perrin
>

Re: High CPU utilization on RHEL5.6/CentOS5.6

On 26 Jul 2011, at 18:58, Christopher Stanton wrote:

> I will see if I can get Perl 5.12 and an associated mod_perl up and
> running on the problem machine and see if the CPU utilization goes
> down.

Is benchmarking out of the question?

Re: High CPU utilization on RHEL5.6/CentOS5.6

Which benchmark module would work best with mod_perl, Perl 5.8 and Perl 5.12?

On Wed, Jul 27, 2011 at 7:22 AM, Dave Hodgkinson wrote:
>
> On 26 Jul 2011, at 18:58, Christopher Stanton wrote:
>
>> I will see if I can get Perl 5.12 and an associated mod_perl up and
>> running on the problem machine and see if the CPU utilization goes
>> down.
>
> Is benchmarking out of the question?
>
>

Re: High CPU utilization on RHEL5.6/CentOS5.6

On Wed, Jul 27, 2011 at 8:36 AM, Christopher Stanton
wrote:
> Which benchmark module would work best with mod_perl, Perl 5.8 and Perl 5.12?

For web apps, I usually write my own test with LWP (or HTTP::Async if
you want to simulate a lot of users). To see where things are slow,
use Devel::NYTProf.

- Perrin

Re: High CPU utilization on RHEL5.6/CentOS5.6

Wait. Is this the one with the horrible bug that knackers OO?

On 25 Jul 2011, at 23:43, Christopher Stanton wrote:

> Requires: perl >= 5.8.8-32