On memory consumption - request for comments

Hi,

there is an ongoing discussion initiated by Max whether Apache::SizeLimit d=
oes=20
the right thing in reporting the current amount of RAM a process does not=20
share with any other process as

unshared_size =3D total_size - shared_size

Max suggests to change that definition to

unshared_size =3D rss - shared_size

Beside the fact that that change should be announced very loudly, perhaps b=
y a=20
new major version, because it requires current installations to adjust thei=
r=20
parameters I am not sure whether it is the right way.

(I am talking about Linux here)

What does that mean?
====================

The total size of a process comprises its complete address space. Normally,=
by=20
far not everything of this space is present in RAM. When the process access=
es=20
a part of its address space that is not present the CPU generates an interr=
upt=20
and the operating system reads in that piece from disk or allocates an empt=
y=20
page and thus makes the accessed page present. Then the operation is repeat=
ed=20
and this time it succeeds. The process normally is not aware of all this.

The part of the process that is really present in RAM is the RSS.

Now, Linux comes with the /proc/$PID/smaps device that reports sizes for=20
shared and private portions of the process' address space.

How does that work?
===================3D

Linux organizes the RAM and address spaces in fixed size chunks, so called=
=20
pages (normally 4kb). Now, a single page of RAM can belongs to only one=20
process or it can be used by multiple processes (for example because they u=
se=20
the same algorithmic part of the C library that is read-only). So, each pag=
e=20
has a reference count. If that refcount is 1 the page is used by only one=20
process and hence private to it. If the refcount is >1 the page is shared=20
among multiple processes.

When /proc/$PID/smaps is read for a process Linux walks all pages of the=20
process and classifies them in 3 groups:

=2D the page is present in RAM and has a refcount==1
==> add it to the process total size and to the private portion

=2D the page is present in RAM and has a refcount>1
==> add it to the process total size and to the shared portion

=2D the page is not present in RAM
==> add it to the process total size

The point here is, for a page that is not present Linux cannot read the=20
refcount because that count is also not present in RAM. So, to decide if a=
=20
page is shared or not it would have to read in the page. This is too expens=
ive=20
an operation only to read the refcount.

So, while in theory a page is either used by only one process and hence=20
private or by multiple and hence shared in practice we have

total_size =3D private + shared + notpresent

where notpresent is either shared or private, we cannot know.

How processes are created?
==================== =====3D=
=3D

Under Linux a process is create by the fork() or clone() system calls. In=20
theory the operating system duplicates the complete address space of the=20
process calling fork. One copy belongs to the original process (the parent)
the other is for the new process (the child).

But if we really had to copy the whole address space fork() would be a real=
ly=20
expensive operation. In fact, only a table holding pointers to the pages th=
at=20
comprise the parent's address space is duplicated. And all pages are marked=
=20
read-only and their reference count is incremented.

Now, if one of the processes wants to write to a page the CPU again generat=
es=20
an interrupt because the page is marked as read-only. The operating system=
=20
catches that interrupt. And only now the actual page is duplicated. One pag=
e=20
for the writing process and one for the others. The refcount of the new pag=
e=20
becomes 1 that of the old is decremented. This working pattern is called co=
py-
on-write.

With the apache web server we have one parent process that spawns many=20
children. At first, almost all of the child's address space is shared with =
the=20
parent due to copy-on-write. Over its lifetime the child's private address=
=20
space grows by 2 means:

* it allocates more memory
==> total_size grows, unshared grows but shared stays the same.
* it writes to portions that were initially shared with the parent
==> unshared grows, shared shrinks but total_size does not change.

Now, what is the goal of Apache::SizeLimit?
==================== =====3D=
==================

If the overall working set of all apache children becomes larger than the=20
available RAM the system then the operating system has to fetch from the di=
sk=20
code and/or data for each request and by doing so it has to evict pages tha=
t=20
will be needed by the next request shortly after.

Apache::SizeLimit (ASL hereafter) tries to avoid this situation.

Note, there is no problem with large process sizes and heavy swap space usa=
ge=20
if the data remains there and is normally not used.

ASL can monitor a few values per apache child, the total size of the proces=
s,=20
the RSS portion, the portion that is reported as shared and the private par=
t.

As for the "unshared" value above it can be defined as:
(all rvalues are reported by /proc/$PID/smaps)

1) unshared =3D size - shared

2) unshared =3D rss - shared

3) unshared =3D private
this is just the same as 2) because rss =3D shared + private

What are the implications?
==================== =====3D=
=3D

1) since size =3D shared + private + notpresent unshared becomes

unshared =3D private + notpresent

Of notpresent we know nothing for sure. It is a mix of shared and private=20
pages.

Now, if an administrator turns off swapping (swapoff /dev/...) the part of=
=20
notpresent that has been there becomes present. So, we can expect shared to=
=20
grow considerably. unshared will shrink by the same amount.

As for absolute values, unshared in this case is quite large a number becau=
se=20
it is on top of notpresent.

2) here unshared lacks the notpresent part. So the actual number is much le=
ss=20
than it would be in case 1.

But if an administrator turns off swapping now a part of notpresent will be=
=20
added to unshared. unshared may suddenly jump over the limit in all apache=
=20
children.

Well, an administrator doing that on a busy web server should be converted=
=20
into a httpd ...

So, I am quite undecided what to do.

Please comment!

See also
http://foertsch.name/ModPerl-Tricks/Measuring-memory-consump tion/index.sh=
tml


What else could be done to hit ASL's goal?
==================== =====3D=
=================3D

There are a few other status fields that can be possibly used:

=2D "Swap" in /proc/$PID/smaps
don't know for sure what that means but sounds good. Need to inspect the
kernel code

=2D "Referenced" in /proc/$PID/smaps
can be used to find out how many RAM a process has accessed since the last
reset of the counter. We could reset it in PerlPostReadRequestHandler and
read in a $r->pool cleanup.

=2D "Pss" in /proc/$PID/smaps
segment size divided by the refcount

=2D "VmSwap" in /proc/$PID/status
for example: terminate if the process starts to use swap space

Certainly more.

Torsten Förtsch

=2D-=20
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net

Re: On memory consumption - request for comments

Torsten Förtsch wrote:
> Hi,
>
> there is an ongoing discussion...

Great "article", Torsten. Thanks.

Re: On memory consumption - request for comments

On 11 Feb 2011, at 14:32, Andr=E9 Warnier wrote:

> Torsten Förtsch wrote:
>> Hi,
>> there is an ongoing discussion...
>=20
> Great "article", Torsten. Thanks.
>=20
>=20


Yes. Please post on the interwebs so I can point colleagues at it.

I think A::SL is the wrong hammer for their nail but I need stronger =
arguments :)

Re: On memory consumption - request for comments

On 02/11/2011 09:26 AM, Torsten Förtsch wrote:

> What does that mean?
> ====================
>
> The total size of a process comprises its complete address space. Normally, by
> far not everything of this space is present in RAM.

I'm not sure I'm understanding you correctly here, but are you saying
that most of the time not all of a process's memory space is in hardware
RAM? If you are saying that, then I would disagree, at least in the
context of httpd/mod_perl. Most of the time (if not all the time) the
goal should be to have enough RAM that your httpd processes completely
fit into memory.

Or am I missing something?

--
Michael Peters
Plus Three, LP

Re: On memory consumption - request for comments

Am Fr, 11.02.2011, 16:46, schrieb Michael Peters:
> On 02/11/2011 09:26 AM, Torsten Förtsch wrote:
>
>> What does that mean?
>> ====================
>>
>> The total size of a process comprises its complete address space.
>> Normally, by
>> far not everything of this space is present in RAM.
>
> I'm not sure I'm understanding you correctly here, but are you saying
> that most of the time not all of a process's memory space is in hardware
> RAM? If you are saying that, then I would disagree, at least in the
> context of httpd/mod_perl. Most of the time (if not all the time) the
> goal should be to have enough RAM that your httpd processes completely
> fit into memory.
>
> Or am I missing something?
>
> --
> Michael Peters
> Plus Three, LP
>

I didnt have time yet to read Torsten's post (will do later) but I will
take a stab at this question. You are missing the difference between
address space and used memory. Sample extract from /proc/*/smaps:

Size: 884 kB
Rss: 536 kB
Pss: 17 kB
Shared_Clean: 536 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 536 kB
Swap: 0 kB

In this case the address space has a size of 884 kb. Only 536 kb are used
though (Rss/Referenced). All of this space is shared. As you can see the
difference between 884 kb and 536 kb is not swapped out but just not
allocated.

Relying on smaps is a little complicated because the format differs with
the kernel version.

I usually use (just) Private_Dirty to determine if an apache process
should be restarted. Its hard to find good thresholds though.

Hendrik

Re: On memory consumption - request for comments

On 02/11/2011 11:01 AM, Hendrik Schumacher wrote:

> I didnt have time yet to read Torsten's post (will do later) but I will
> take a stab at this question. You are missing the difference between
> address space and used memory. Sample extract from /proc/*/smaps:
>
> Size: 884 kB
> Rss: 536 kB
> Pss: 17 kB
> Shared_Clean: 536 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 0 kB
> Referenced: 536 kB
> Swap: 0 kB
>
> In this case the address space has a size of 884 kb. Only 536 kb are used
> though (Rss/Referenced). All of this space is shared. As you can see the
> difference between 884 kb and 536 kb is not swapped out but just not
> allocated.

Interesting. I didn't know that. But I think the questions that Torsten
was posing about what would happen if an admin turned off swap while
things were running doesn't apply then, right? This memory isn't in
swap, it's just not in RAM. So turning off swap won't cause the
shared/unshared sizes to blow up, right?

--
Michael Peters
Plus Three, LP

Re: On memory consumption - request for comments

Am Fr, 11.02.2011, 17:10, schrieb Michael Peters:
> On 02/11/2011 11:01 AM, Hendrik Schumacher wrote:
>
>> I didnt have time yet to read Torsten's post (will do later) but I will
>> take a stab at this question. You are missing the difference between
>> address space and used memory. Sample extract from /proc/*/smaps:
>>
>> Size: 884 kB
>> Rss: 536 kB
>> Pss: 17 kB
>> Shared_Clean: 536 kB
>> Shared_Dirty: 0 kB
>> Private_Clean: 0 kB
>> Private_Dirty: 0 kB
>> Referenced: 536 kB
>> Swap: 0 kB
>>
>> In this case the address space has a size of 884 kb. Only 536 kb are
>> used
>> though (Rss/Referenced). All of this space is shared. As you can see the
>> difference between 884 kb and 536 kb is not swapped out but just not
>> allocated.
>
> Interesting. I didn't know that. But I think the questions that Torsten
> was posing about what would happen if an admin turned off swap while
> things were running doesn't apply then, right? This memory isn't in
> swap, it's just not in RAM. So turning off swap won't cause the
> shared/unshared sizes to blow up, right?
>
> --
> Michael Peters
> Plus Three, LP
>

Yes and no. If I modify the example from above:

Size: 884 kB
Rss: 536 kB
Pss: 17 kB
Shared_Clean: 536 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 536 kB
Swap: 64 kB

then switching off the swap would cause the Shared or the Private to "blow
up" by 64 kb. Only the address space that is neither present nor swapped
does not cause any effect when switching off the swap.

Re: On memory consumption - request for comments

Hi Torsten,

Thanks for the thorough explanation.

I used to be a big proponent of ASL, but I rely on it less since years
ago when you pointed out that the shared sizes were not accurate on
Linux. I know that Smaps helps with that, but it seems fairly
expensive so I've avoided it.

These days I like to use a reasonable MaxRequestsPerChild (e.g. 100)
combined with a fairly high size limit in ASL. That just helps to
catch any unusual growth in a process.

As for what to do with ASL, I think it probably does make more sense
to use RSS since we're trying to avoid swapping, but shared is not
reliable enough for me to trust anymore. I don't think that an admin
swtching off swap while a server is live is worth worrying about, and
people already have to do their own measurement and tuning to choose
sizes when setting this up.

- Perrin

2011/2/11 Torsten Förtsch :
> Hi,
>
> there is an ongoing discussion initiated by Max whether Apache::SizeLimit=
does
> the right thing in reporting the current amount of RAM a process does not
> share with any other process as
>
> =A0unshared_size =3D total_size - shared_size
>
> Max suggests to change that definition to
>
> =A0unshared_size =3D rss - shared_size
>
> Beside the fact that that change should be announced very loudly, perhaps=
by a
> new major version, because it requires current installations to adjust th=
eir
> parameters I am not sure whether it is the right way.
>
> (I am talking about Linux here)
>
> What does that mean?
> ====================
>
> The total size of a process comprises its complete address space. Normall=
y, by
> far not everything of this space is present in RAM. When the process acce=
sses
> a part of its address space that is not present the CPU generates an inte=
rrupt
> and the operating system reads in that piece from disk or allocates an em=
pty
> page and thus makes the accessed page present. Then the operation is repe=
ated
> and this time it succeeds. The process normally is not aware of all this.
>
> The part of the process that is really present in RAM is the RSS.
>
> Now, Linux comes with the /proc/$PID/smaps device that reports sizes for
> shared and private portions of the process' address space.
>
> How does that work?
> ===================3D
>
> Linux organizes the RAM and address spaces in fixed size chunks, so calle=
d
> pages (normally 4kb). Now, a single page of RAM can belongs to only one
> process or it can be used by multiple processes (for example because they=
use
> the same algorithmic part of the C library that is read-only). So, each p=
age
> has a reference count. If that refcount is 1 the page is used by only one
> process and hence private to it. If the refcount is >1 the page is shared
> among multiple processes.
>
> When /proc/$PID/smaps is read for a process Linux walks all pages of the
> process and classifies them in 3 groups:
>
> - the page is present in RAM and has a refcount==1
> =A0  ==3D> add it to the process total size and to the private portio=
n
>
> - the page is present in RAM and has a refcount>1
> =A0  ==3D> add it to the process total size and to the shared portion
>
> - the page is not present in RAM
> =A0  ==3D> add it to the process total size
>
> The point here is, for a page that is not present Linux cannot read the
> refcount because that count is also not present in RAM. So, to decide if =
a
> page is shared or not it would have to read in the page. This is too expe=
nsive
> an operation only to read the refcount.
>
> So, while in theory a page is either used by only one process and hence
> private or by multiple and hence shared in practice we have
>
> =A0total_size =3D private + shared + notpresent
>
> where notpresent is either shared or private, we cannot know.
>
> How processes are created?
> ==================== =====
==
>
> Under Linux a process is create by the fork() or clone() system calls. In
> theory the operating system duplicates the complete address space of the
> process calling fork. One copy belongs to the original process (the paren=
t)
> the other is for the new process (the child).
>
> But if we really had to copy the whole address space fork() would be a re=
ally
> expensive operation. In fact, only a table holding pointers to the pages =
that
> comprise the parent's address space is duplicated. And all pages are mark=
ed
> read-only and their reference count is incremented.
>
> Now, if one of the processes wants to write to a page the CPU again gener=
ates
> an interrupt because the page is marked as read-only. The operating syste=
m
> catches that interrupt. And only now the actual page is duplicated. One p=
age
> for the writing process and one for the others. The refcount of the new p=
age
> becomes 1 that of the old is decremented. This working pattern is called =
copy-
> on-write.
>
> With the apache web server we have one parent process that spawns many
> children. At first, almost all of the child's address space is shared wit=
h the
> parent due to copy-on-write. Over its lifetime the child's private addres=
s
> space grows by 2 means:
>
> * it allocates more memory
>  ==3D> total_size grows, unshared grows but shared stays the same.
> * it writes to portions that were initially shared with the parent
>  ==3D> unshared grows, shared shrinks but total_size does not change.
>
> Now, what is the goal of Apache::SizeLimit?
> ==================== =====
===================3D
>
> If the overall working set of all apache children becomes larger than the
> available RAM the system then the operating system has to fetch from the =
disk
> code and/or data for each request and by doing so it has to evict pages t=
hat
> will be needed by the next request shortly after.
>
> Apache::SizeLimit (ASL hereafter) tries to avoid this situation.
>
> Note, there is no problem with large process sizes and heavy swap space u=
sage
> if the data remains there and is normally not used.
>
> ASL can monitor a few values per apache child, the total size of the proc=
ess,
> the RSS portion, the portion that is reported as shared and the private p=
art.
>
> As for the "unshared" value above it can be defined as:
> (all rvalues are reported by /proc/$PID/smaps)
>
> =A01) unshared =3D size - shared
>
> =A02) unshared =3D rss - shared
>
> =A03) unshared =3D private
> =A0 =A0this is just the same as 2) because rss =3D shared + private
>
> What are the implications?
> ==================== =====
==
>
> 1) since size =3D shared + private + notpresent unshared becomes
>
> =A0unshared =3D private + notpresent
>
> Of notpresent we know nothing for sure. It is a mix of shared and private
> pages.
>
> Now, if an administrator turns off swapping (swapoff /dev/...) the part o=
f
> notpresent that has been there becomes present. So, we can expect shared =
to
> grow considerably. unshared will shrink by the same amount.
>
> As for absolute values, unshared in this case is quite large a number bec=
ause
> it is on top of notpresent.
>
> 2) here unshared lacks the notpresent part. So the actual number is much =
less
> than it would =A0be in case 1.
>
> But if an administrator turns off swapping now a part of notpresent will =
be
> added to unshared. unshared may suddenly jump over the limit in all apach=
e
> children.
>
> Well, an administrator doing that on a busy web server should be converte=
d
> into a httpd ...
>
> So, I am quite undecided what to do.
>
> Please comment!
>
> See also
> =A0http://foertsch.name/ModPerl-Tricks/Measuring-memory-cons umption/index=
..shtml
>
>
> What else could be done to hit ASL's goal?
> ==================== =====
==================
>
> There are a few other status fields that can be possibly used:
>
> - "Swap" in /proc/$PID/smaps
> =A0don't know for sure what that means but sounds good. Need to inspect t=
he
> =A0kernel code
>
> - "Referenced" in /proc/$PID/smaps
> =A0can be used to find out how many RAM a process has accessed since the =
last
> =A0reset of the counter. We could reset it in PerlPostReadRequestHandler =
and
> =A0read in a $r->pool cleanup.
>
> - "Pss" in /proc/$PID/smaps
> =A0segment size divided by the refcount
>
> - "VmSwap" in /proc/$PID/status
> =A0for example: terminate if the process starts to use swap space
>
> Certainly more.
>
> Torsten Förtsch
>
> --
> Need professional modperl support? Hire me! (http://foertsch.name)
>
> Like fantasy? http://kabatinte.net
>

Re: On memory consumption - request for comments

Hi,

I would go with rss - shared_size. Especially on 64bit-platforms the
total_size gives much too high values (even without swap space). Using the
other values like Pss or Swap is not possible on older kernels (I don't
have these values on EC2-instances for example). An option would be to
substract Swap from unshared_size if it is present. Personally I don't
bother if swapped space is shared or not.

Hendrik

Am Fr, 11.02.2011, 15:26, schrieb Torsten Förtsch:
> Hi,
>
> there is an ongoing discussion initiated by Max whether Apache::SizeLimit
> does
> the right thing in reporting the current amount of RAM a process does not
> share with any other process as
>
> unshared_size = total_size - shared_size
>
> Max suggests to change that definition to
>
> unshared_size = rss - shared_size
>
> Beside the fact that that change should be announced very loudly, perhaps
> by a
> new major version, because it requires current installations to adjust
> their
> parameters I am not sure whether it is the right way.
>
> (I am talking about Linux here)
>
> What does that mean?
> ====================
>
> The total size of a process comprises its complete address space.
> Normally, by
> far not everything of this space is present in RAM. When the process
> accesses
> a part of its address space that is not present the CPU generates an
> interrupt
> and the operating system reads in that piece from disk or allocates an
> empty
> page and thus makes the accessed page present. Then the operation is
> repeated
> and this time it succeeds. The process normally is not aware of all this.
>
> The part of the process that is really present in RAM is the RSS.
>
> Now, Linux comes with the /proc/$PID/smaps device that reports sizes for
> shared and private portions of the process' address space.
>
> How does that work?
> ===================
>
> Linux organizes the RAM and address spaces in fixed size chunks, so called
> pages (normally 4kb). Now, a single page of RAM can belongs to only one
> process or it can be used by multiple processes (for example because they
> use
> the same algorithmic part of the C library that is read-only). So, each
> page
> has a reference count. If that refcount is 1 the page is used by only one
> process and hence private to it. If the refcount is >1 the page is shared
> among multiple processes.
>
> When /proc/$PID/smaps is read for a process Linux walks all pages of the
> process and classifies them in 3 groups:
>
> - the page is present in RAM and has a refcount==1
> ==> add it to the process total size and to the private portion
>
> - the page is present in RAM and has a refcount>1
> ==> add it to the process total size and to the shared portion
>
> - the page is not present in RAM
> ==> add it to the process total size
>
> The point here is, for a page that is not present Linux cannot read the
> refcount because that count is also not present in RAM. So, to decide if a
> page is shared or not it would have to read in the page. This is too
> expensive
> an operation only to read the refcount.
>
> So, while in theory a page is either used by only one process and hence
> private or by multiple and hence shared in practice we have
>
> total_size = private + shared + notpresent
>
> where notpresent is either shared or private, we cannot know.
>
> How processes are created?
> ==========================
>
> Under Linux a process is create by the fork() or clone() system calls. In
> theory the operating system duplicates the complete address space of the
> process calling fork. One copy belongs to the original process (the
> parent)
> the other is for the new process (the child).
>
> But if we really had to copy the whole address space fork() would be a
> really
> expensive operation. In fact, only a table holding pointers to the pages
> that
> comprise the parent's address space is duplicated. And all pages are
> marked
> read-only and their reference count is incremented.
>
> Now, if one of the processes wants to write to a page the CPU again
> generates
> an interrupt because the page is marked as read-only. The operating system
> catches that interrupt. And only now the actual page is duplicated. One
> page
> for the writing process and one for the others. The refcount of the new
> page
> becomes 1 that of the old is decremented. This working pattern is called
> copy-
> on-write.
>
> With the apache web server we have one parent process that spawns many
> children. At first, almost all of the child's address space is shared with
> the
> parent due to copy-on-write. Over its lifetime the child's private address
> space grows by 2 means:
>
> * it allocates more memory
> ==> total_size grows, unshared grows but shared stays the same.
> * it writes to portions that were initially shared with the parent
> ==> unshared grows, shared shrinks but total_size does not change.
>
> Now, what is the goal of Apache::SizeLimit?
> ===========================================
>
> If the overall working set of all apache children becomes larger than the
> available RAM the system then the operating system has to fetch from the
> disk
> code and/or data for each request and by doing so it has to evict pages
> that
> will be needed by the next request shortly after.
>
> Apache::SizeLimit (ASL hereafter) tries to avoid this situation.
>
> Note, there is no problem with large process sizes and heavy swap space
> usage
> if the data remains there and is normally not used.
>
> ASL can monitor a few values per apache child, the total size of the
> process,
> the RSS portion, the portion that is reported as shared and the private
> part.
>
> As for the "unshared" value above it can be defined as:
> (all rvalues are reported by /proc/$PID/smaps)
>
> 1) unshared = size - shared
>
> 2) unshared = rss - shared
>
> 3) unshared = private
> this is just the same as 2) because rss = shared + private
>
> What are the implications?
> ==========================
>
> 1) since size = shared + private + notpresent unshared becomes
>
> unshared = private + notpresent
>
> Of notpresent we know nothing for sure. It is a mix of shared and private
> pages.
>
> Now, if an administrator turns off swapping (swapoff /dev/...) the part of
> notpresent that has been there becomes present. So, we can expect shared
> to
> grow considerably. unshared will shrink by the same amount.
>
> As for absolute values, unshared in this case is quite large a number
> because
> it is on top of notpresent.
>
> 2) here unshared lacks the notpresent part. So the actual number is much
> less
> than it would be in case 1.
>
> But if an administrator turns off swapping now a part of notpresent will
> be
> added to unshared. unshared may suddenly jump over the limit in all apache
> children.
>
> Well, an administrator doing that on a busy web server should be converted
> into a httpd ...
>
> So, I am quite undecided what to do.
>
> Please comment!
>
> See also
> http://foertsch.name/ModPerl-Tricks/Measuring-memory-consump tion/index.shtml
>
>
> What else could be done to hit ASL's goal?
> ==========================================
>
> There are a few other status fields that can be possibly used:
>
> - "Swap" in /proc/$PID/smaps
> don't know for sure what that means but sounds good. Need to inspect the
> kernel code
>
> - "Referenced" in /proc/$PID/smaps
> can be used to find out how many RAM a process has accessed since the
> last
> reset of the counter. We could reset it in PerlPostReadRequestHandler
> and
> read in a $r->pool cleanup.
>
> - "Pss" in /proc/$PID/smaps
> segment size divided by the refcount
>
> - "VmSwap" in /proc/$PID/status
> for example: terminate if the process starts to use swap space
>
> Certainly more.
>
> Torsten Förtsch
>
> --
> Need professional modperl support? Hire me! (http://foertsch.name)
>
> Like fantasy? http://kabatinte.net
>

Re: On memory consumption - request for comments

On Friday, February 11, 2011 17:20:07 Perrin Harkins wrote:
> I used to be a big proponent of ASL, but I rely on it less since years
> ago when you pointed out that the shared sizes were not accurate on
> Linux. I know that Smaps helps with that, but it seems fairly
> expensive so I've avoided it.

To assess how expensive it is, here is a comparison. I compare reading 2492=
7=20
bytes from /proc/$$/smaps, from a plain file and from /dev/urandom the pseu=
do=20
random number generator in kernel:

$ perl -MBenchmark=3D:all,:hireswallclock -e '
cmpthese timethese -10, {
file=3D>sub{open my $f, "<", "xx" or die; $/=3D\24927;
24927==length readline $f or die},
smaps=3D>sub{open my $f, "<", "/proc/".$ARGV[0]."/smaps" or die; $/=3D\2=
4927;
24927==length readline $f or die},
urandom=3D>sub{open my $f, "<", "/dev/urandom" or die; $/=3D\24927;
24927==length readline $f or die}}' $$
Benchmark: running file, smaps, urandom for at least 10 CPU seconds...
file: 12.4656 wallclock secs ( 7.94 usr + 4.45 sys =3D 12.39 CPU) @=
=20
47120.02/s (n=3D583817)
smaps: 10.1048 wallclock secs ( 0.59 usr + 9.50 sys =3D 10.09 CPU) @=
=20
3287.22/s (n=3D33168)
urandom: 11.618 wallclock secs ( 0.03 usr + 11.57 sys =3D 11.60 CPU) @=20
198.28/s (n=3D2300)
Rate urandom smaps file
urandom 198/s -- -94% -100%
smaps 3287/s 1558% -- -93%
file 47120/s 23665% 1333% --

Torsten Förtsch

=2D-=20
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net

Re: On memory consumption - request for comments

On Friday, February 11, 2011 17:19:04 Hendrik Schumacher wrote:
> > Interesting. I didn't know that. But I think the questions that Torsten
> > was posing about what would happen if an admin turned off swap while
> > things were running doesn't apply then, right? This memory isn't in
> > swap, it's just not in RAM. So turning off swap won't cause the
> > shared/unshared sizes to blow up, right?
>=20
> Yes and no. If I modify the example from above:
>=20
> Size: 884 kB
> Rss: 536 kB
> Pss: 17 kB
> Shared_Clean: 536 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 0 kB
> Referenced: 536 kB
> Swap: 64 kB
>=20
> then switching off the swap would cause the Shared or the Private to "blow
> up" by 64 kb. Only the address space that is neither present nor swapped
> does not cause any effect when switching off the swap.

Maybe I oversimplified a bit here. Turning off swap was also only an exampl=
e=20
of "a large part of notpresent suddenly becomes known". But actually I can'=
t=20
think of another example without activity of the process in question. And i=
f=20
the process is the culprit of the unusual event then it is perhaps also wor=
th=20
to be terminated.

Torsten Förtsch

=2D-=20
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net

Re: On memory consumption - request for comments

On Friday, February 11, 2011 17:20:07 Perrin Harkins wrote:
> These days I like to use a reasonable MaxRequestsPerChild

same here. I try to figure out the worst case before going life. Then=20
MaxClients can be set accordingly. Plus a reasonable MaxRequestsPerChild an=
d=20
you are done.

Also, these days RAM is not the bottleneck as it was 10 years ago. In a=20
project I had a 8GB box that hit the CPU limit at 250 active parallel=20
requests. But it used only slightly more than half of the RAM.

Torsten Förtsch

=2D-=20
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net

Re: On memory consumption - request for comments

2011/2/11 Torsten Förtsch :
> Also, these days RAM is not the bottleneck as it was 10 years ago.

Agreed. I used to spend lots of time trying to reduce memory used by
mod_perl projects, but now I don't even look at it unless I'm running
into trouble.

- Perrin

Re: On memory consumption - request for comments

2011/2/11 Torsten Förtsch :
> On Friday, February 11, 2011 17:20:07 Perrin Harkins wrote:
>> These days I like to use a reasonable MaxRequestsPerChild
>
> same here. I try to figure out the worst case before going life. Then
> MaxClients can be set accordingly. Plus a reasonable MaxRequestsPerChild =
and
> you are done.
>
> Also, these days RAM is not the bottleneck as it was 10 years ago. In a
> project I had a 8GB box that hit the CPU limit at 250 active parallel
> requests. But it used only slightly more than half of the RAM.

I don't think there is a danger of running into swap with today's
environments, so I wondered about why Sizelimit deals with swap
metrics. If you do run into swap with a box with 8 gigs of ram, swap
will probably not save you, as you'll be swapping out large processes.
I keep my MaxRequestsPerChild low also, but also like to use
SizeLimit to cap the maximum process size.

Re: On memory consumption - request for comments

On 02/11/2011 06:26 AM, Torsten Förtsch wrote:
> Now, if an administrator turns off swapping (swapoff /dev/...) the part of
> notpresent that has been there becomes present. So, we can expect shared to
> grow considerably. unshared will shrink by the same amount.
>
> But if an administrator turns off swapping now a part of notpresent will be
> added to unshared. unshared may suddenly jump over the limit in all apache
> children.

Hey Torsten. First of all, wow, thank you for this awesome write-up.
That was probably the best overview of copy-on-write and page types I've
ever read.

I think that if an administrator turns off the swap, they would in fact
*want* ASL to behave such that it killed httpds that were taking up too
much physical RAM, even if that happened suddenly. The space for mapping
pages to has suddenly decreased drastically, so it makes sense to
suddenly be much more aggressive about killing processes.

> There are a few other status fields that can be possibly used:
>
> - "Swap" in /proc/$PID/smaps
> don't know for sure what that means but sounds good. Need to inspect the
> kernel code

Hmm, if that does actually count the size of mapped swap pages, then we
could possibly do rss + swap as the "size".

-Max
--
http://www.bugzillasource.com/
Competent, Friendly Bugzilla, Perl, and IT Services

Re: On memory consumption - request for comments

Hey Fred. So given the discussion that we've had on this, do you think
that the next version of SizeLimit could change its Linux behavior to
return the more appropriate rss size?

-Max
--
http://www.bugzillasource.com/
Competent, Friendly Bugzilla, Perl, and IT Services

Re: On memory consumption - request for comments

On Thursday, February 24, 2011 07:45:29 Max Kanat-Alexander wrote:
> Hey Fred. So given the discussion that we've had on this, do you think
> that the next version of SizeLimit could change its Linux behavior to
> return the more appropriate rss size?

yes

Torsten Förtsch

=2D-=20
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net

Re: On memory consumption - request for comments

On 02/23/2011 11:40 PM, Torsten Förtsch wrote:
> On Thursday, February 24, 2011 07:45:29 Max Kanat-Alexander wrote:
>> Hey Fred. So given the discussion that we've had on this, do you think
>> that the next version of SizeLimit could change its Linux behavior to
>> return the more appropriate rss size?
>
> yes

Awesome, thank you. :-)

-Max
--
http://www.bugzillasource.com/
Competent, Friendly Bugzilla, Perl, and IT Services

Re: On memory consumption - request for comments

On Thursday, February 24, 2011 09:29:23 Max Kanat-Alexander wrote:
> On 02/23/2011 11:40 PM, Torsten Förtsch wrote:
> > On Thursday, February 24, 2011 07:45:29 Max Kanat-Alexander wrote:
> >> Hey Fred. So given the discussion that we've had on this, do you
> >> think that the next version of SizeLimit could change its Linux
> >> behavior to return the more appropriate rss size?
> >
> >=20
> >
> > yes
>=20
> Awesome, thank you. :-)

Max, could you please check if the following patch does what you want and t=
ry=20
it out in your environment?

Index: lib/Apache/SizeLimit/Core.pm
==================== =====3D=
==================== =====3D=
=================3D
=2D-- lib/Apache/SizeLimit/Core.pm (revision 1069512)
+++ lib/Apache/SizeLimit/Core.pm (revision 1069513)
@@ -117,9 +117,9 @@
sub _check_size {
my $class =3D shift;
=20
=2D my ($size, $share) =3D $class->_platform_check_size();
+ my ($size, $share, $unshared) =3D $class->_platform_check_size();
=20
=2D return ($size, $share, $size - $share);
+ return ($size, $share, defined $unshared ? $unshared : $size - $share);
}
=20
sub _load {
@@ -176,7 +176,9 @@
return $class->_linux_size_check() unless $USE_SMAPS;
=20
my $s =3D Linux::Smaps->new($$)->all;
=2D return ($s->size, $s->shared_clean + $s->shared_dirty);
+ return ($s->size,
+ $s->shared_clean + $s->shared_dirty,
+ $s->private_clean + $s->private_dirty);
}
=20
sub _linux_size_check {


Torsten Förtsch

=2D-=20
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net

Re: On memory consumption - request for comments

On 02/24/2011 12:41 AM, Torsten Förtsch wrote:
> Max, could you please check if the following patch does what you want and try
> it out in your environment?

Hey Torsten. Yes, I tested it and it works. Thank you so much! :-)

-Max
--
http://www.bugzillasource.com/
Competent, Friendly Bugzilla, Perl, and IT Services

Re: On memory consumption - request for comments

What's the rough ETA before this makes it to CPAN?

Working out if I want to add the patch to our build process or not...

On 24 Feb 2011, at 08:41, Torsten Förtsch wrote:

> On Thursday, February 24, 2011 09:29:23 Max Kanat-Alexander wrote:
>> On 02/23/2011 11:40 PM, Torsten Förtsch wrote:
>>> On Thursday, February 24, 2011 07:45:29 Max Kanat-Alexander wrote:
>>>> Hey Fred. So given the discussion that we've had on this, do =
you
>>>> think that the next version of SizeLimit could change its Linux
>>>> behavior to return the more appropriate rss size?
>>>=20
>>>=20
>>>=20
>>> yes
>>=20
>> Awesome, thank you. :-)
>=20
> Max, could you please check if the following patch does what you want =
and try=20
> it out in your environment?
>=20
> Index: lib/Apache/SizeLimit/Core.pm
> ==================== =====
==================== =====3D=
==================
> --- lib/Apache/SizeLimit/Core.pm (revision 1069512)
> +++ lib/Apache/SizeLimit/Core.pm (revision 1069513)
> @@ -117,9 +117,9 @@
> sub _check_size {
> my $class =3D shift;
>=20
> - my ($size, $share) =3D $class->_platform_check_size();
> + my ($size, $share, $unshared) =3D $class->_platform_check_size();
>=20
> - return ($size, $share, $size - $share);
> + return ($size, $share, defined $unshared ? $unshared : $size - =
$share);
> }
>=20
> sub _load {
> @@ -176,7 +176,9 @@
> return $class->_linux_size_check() unless $USE_SMAPS;
>=20
> my $s =3D Linux::Smaps->new($$)->all;
> - return ($s->size, $s->shared_clean + $s->shared_dirty);
> + return ($s->size,
> + $s->shared_clean + $s->shared_dirty,
> + $s->private_clean + $s->private_dirty);
> }
>=20
> sub _linux_size_check {
>=20
>=20
> Torsten Förtsch
>=20
> --=20
> Need professional modperl support? Hire me! (http://foertsch.name)
>=20
> Like fantasy? http://kabatinte.net

Re: On memory consumption - request for comments

On Thursday, February 24, 2011 11:27:42 Dave Hodgkinson wrote:
> What's the rough ETA before this makes it to CPAN?

end of April, perhaps.

Torsten Förtsch

=2D-=20
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net

Re: On memory consumption - request for comments

2011/2/24 Torsten Förtsch :
> On Thursday, February 24, 2011 11:27:42 Dave Hodgkinson wrote:
>> What's the rough ETA before this makes it to CPAN?
>
> end of April, perhaps.

We can release this earlier than 2.0.6 if there are enough testers to
give feedback.

Re: On memory consumption - request for comments

On 24 Feb 2011, at 15:44, Fred Moyer wrote:

> 2011/2/24 Torsten Förtsch :
>> On Thursday, February 24, 2011 11:27:42 Dave Hodgkinson wrote:
>>> What's the rough ETA before this makes it to CPAN?
>>=20
>> end of April, perhaps.
>=20
> We can release this earlier than 2.0.6 if there are enough testers to
> give feedback.

I'll ask here.=