Memory investigation

Hi folks. I have a complicated PHP app that is eating up more memory than I
think it should. I have a couple of theories as to where it could be going,
but I need some way to verify it.

There are a number of large data structures (mostly arrays) that get built up
throughout the course of the request. What I'd like to be able to do is at
certain points check to see how much memory those data structures are using.
Because they're not all built at once, the usual "check memory before, build,
check after" routine won't work. Plus, that gets screwed up by PHP's use of
copy-on-write at times.

I know that it would result in essentially over-reporting, but I would ideally
like to be able to ignore the copy-on-write issue and say "if this variable
were the only thing in memory (and its dependents, of course, for a nested
array), what would its memory usage be? I just have no idea how to do that.

Anyone have a suggestion for how to accomplish that?

--Larry Garfield

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Memory investigation

ON Linux I have kcacheGrind setup with Xdebug and I find it is a nice
little thing to have. It won't tell you the memory consumed but it will
find cycles and display object maps. if you have Kcachegrind it is
likely you have valgrind installed.
http://www2.mandriva.com/
http://valgrind.org/
http://kcachegrind.sourceforge.net/html/Home.html
http://xdebug.org/
Larry Garfield wrote:
> Hi folks. I have a complicated PHP app that is eating up more memory than I
> think it should. I have a couple of theories as to where it could be going,
> but I need some way to verify it.
>
> There are a number of large data structures (mostly arrays) that get built up
> throughout the course of the request. What I'd like to be able to do is at
> certain points check to see how much memory those data structures are using.
> Because they're not all built at once, the usual "check memory before, build,
> check after" routine won't work. Plus, that gets screwed up by PHP's use of
> copy-on-write at times.
>
> I know that it would result in essentially over-reporting, but I would ideally
> like to be able to ignore the copy-on-write issue and say "if this variable
> were the only thing in memory (and its dependents, of course, for a nested
> array), what would its memory usage be? I just have no idea how to do that.
>
> Anyone have a suggestion for how to accomplish that?
>
> --Larry Garfield

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Memory investigation

Yep, I'm familiar with XDebug and KCacheGrind. As you say, though, it
doens't (as far as I am aware) offer the particular data that I'm after.
We've already got cachegrind gurus working on the code who know how to
use it better than I do. :-) What I'm looking for is "see this big
cache object we've been building up in memory for this whole page
request? Now that we're done with the page, how big is it, really?"

--Larry Garfield

user@domain.invalid wrote:
> ON Linux I have kcacheGrind setup with Xdebug and I find it is a nice
> little thing to have. It won't tell you the memory consumed but it will
> find cycles and display object maps. if you have Kcachegrind it is
> likely you have valgrind installed.
> http://www2.mandriva.com/
> http://valgrind.org/
> http://kcachegrind.sourceforge.net/html/Home.html
> http://xdebug.org/
> Larry Garfield wrote:
>> Hi folks. I have a complicated PHP app that is eating up more memory
>> than I think it should. I have a couple of theories as to where it
>> could be going, but I need some way to verify it.
>>
>> There are a number of large data structures (mostly arrays) that get
>> built up throughout the course of the request. What I'd like to be
>> able to do is at certain points check to see how much memory those
>> data structures are using. Because they're not all built at once, the
>> usual "check memory before, build, check after" routine won't work.
>> Plus, that gets screwed up by PHP's use of copy-on-write at times.
>>
>> I know that it would result in essentially over-reporting, but I would
>> ideally like to be able to ignore the copy-on-write issue and say "if
>> this variable were the only thing in memory (and its dependents, of
>> course, for a nested array), what would its memory usage be? I just
>> have no idea how to do that.
>>
>> Anyone have a suggestion for how to accomplish that?
>>
>> --Larry Garfield
>

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Memory investigation

On 3 March 2010 15:49, larry@garfieldtech.com wrot=
e:

> Yep, I'm familiar with XDebug and KCacheGrind.  As you say, though, =
it
> doens't (as far as I am aware) offer the particular data that I'm after.
>  We've already got cachegrind gurus working on the code who know how=
to use
> it better than I do. :-)  What I'm looking for is "see this big cach=
e object
> we've been building up in memory for this whole page request?  Now t=
hat
> we're done with the page, how big is it, really?"

You could try Memtrack (http://pecl.php.net/package/memtrack) but I
can't comment as to what state it's in.

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Memory investigation

function check_memory_usage(&$memory)
{
$memory[] = memory_get_usage();
return $memory;
}

something like this?
you can put it wherever you like and returns an array for further
processing. You could optionally add a second argument to set the index
to a name and check if the name exists to add 1 to the end of the name
so your indexes stay maintained.

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Memory investigation

That's not really what I'm after. Let me try an example:

function foo($id) {
static $foos = array();

if (empty($foos[$id]) {
$foos[$id] = load_foo($id);
}
return $foos[$id];
}

When load_foo() is slow (e.g., lots of DB traffic or remote-server calls
or whatever), such caching can have a significant performance boost.
Sometime after foo() has been called 15 times from 30 places in code,
when I get to the end of the request (or just every time I call foo()
would be fine) I want to be able to do something like:

$cost = get_memory_used_by($foos);

So that I can determine how much memory that caching is costing me over
the lifetime of the page, and determine if it's a worthwhile trade-off.

--Larry Garfield

dsiembab01@gmail.com wrote:
> function check_memory_usage(&$memory)
> {
> $memory[] = memory_get_usage();
> return $memory;
> }
>
> something like this?
> you can put it wherever you like and returns an array for further
> processing. You could optionally add a second argument to set the index
> to a name and check if the name exists to add 1 to the end of the name
> so your indexes stay maintained.
>

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Memory investigation

global $fooSize =3D 0;

function foo($id) {
.....
global $fooSize;
if (empty($foos($id)) {
$b =3D get_memory_usage(true);
$foos[$id] =3D load_foo($id);
$fooSize+=3D $b - get_memory_usage(true);
}
....
}
On Wed, Mar 3, 2010 at 8:16 PM, larry@garfieldtech.com
wrote:
> That's not really what I'm after. =A0Let me try an example:
>
> function foo($id) {
> =A0static $foos =3D array();
>
> =A0if (empty($foos[$id]) {
> =A0 =A0$foos[$id] =3D load_foo($id);
> =A0}
> =A0return $foos[$id];
> }
>
> When load_foo() is slow (e.g., lots of DB traffic or remote-server calls =
or
> whatever), such caching can have a significant performance boost. Sometim=
e
> after foo() has been called 15 times from 30 places in code, when I get t=
o
> the end of the request (or just every time I call foo() would be fine) I
> want to be able to do something like:
>
> $cost =3D get_memory_used_by($foos);
>
> So that I can determine how much memory that caching is costing me over t=
he
> lifetime of the page, and determine if it's a worthwhile trade-off.
>
> --Larry Garfield
>
> dsiembab01@gmail.com wrote:
>>
>> function check_memory_usage(&$memory)
>> {
>> =A0 =A0$memory[] =3D memory_get_usage();
>> =A0 =A0return $memory;
>> }
>>
>> something like this?
>> you can put it wherever you like and returns an array for further
>> processing. You could optionally add a second argument to set the index =
to a
>> name and check if the name exists to add 1 to the end of the name so you=
r
>> indexes stay maintained.
>>
>
> --
> PHP General Mailing List (http://www.php.net/)
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Memory investigation

couple questions Larry is this application composed of classes or
straight up no holes barred procedural code?

larry@garfieldtech.com wrote:
> That's not really what I'm after. Let me try an example:
>
> function foo($id) {
> static $foos = array();
>
> if (empty($foos[$id]) {
> $foos[$id] = load_foo($id);
> }
> return $foos[$id];
> }
>
> When load_foo() is slow (e.g., lots of DB traffic or remote-server calls
> or whatever), such caching can have a significant performance boost.
> Sometime after foo() has been called 15 times from 30 places in code,
> when I get to the end of the request (or just every time I call foo()
> would be fine) I want to be able to do something like:
>
> $cost = get_memory_used_by($foos);
>
> So that I can determine how much memory that caching is costing me over
> the lifetime of the page, and determine if it's a worthwhile trade-off.
>
> --Larry Garfield
>
> dsiembab01@gmail.com wrote:
>> function check_memory_usage(&$memory)
>> {
>> $memory[] = memory_get_usage();
>> return $memory;
>> }
>>
>> something like this?
>> you can put it wherever you like and returns an array for further
>> processing. You could optionally add a second argument to set the
>> index to a name and check if the name exists to add 1 to the end of
>> the name so your indexes stay maintained.
>>

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Memory investigation

Currently it's mostly procedural with some components that are OO. I
suspect most of the memory sinks are large arrays (we have a lot of
them), but I am not certain of that. Hence my interest in more accurate
investigation tools.

--Larry Garfield

dsiembab01@gmail.com wrote:
> couple questions Larry is this application composed of classes or
> straight up no holes barred procedural code?
>
> larry@garfieldtech.com wrote:
>> That's not really what I'm after. Let me try an example:
>>
>> function foo($id) {
>> static $foos = array();
>>
>> if (empty($foos[$id]) {
>> $foos[$id] = load_foo($id);
>> }
>> return $foos[$id];
>> }
>>
>> When load_foo() is slow (e.g., lots of DB traffic or remote-server
>> calls or whatever), such caching can have a significant performance
>> boost. Sometime after foo() has been called 15 times from 30 places in
>> code, when I get to the end of the request (or just every time I call
>> foo() would be fine) I want to be able to do something like:
>>
>> $cost = get_memory_used_by($foos);
>>
>> So that I can determine how much memory that caching is costing me
>> over the lifetime of the page, and determine if it's a worthwhile
>> trade-off.
>>
>> --Larry Garfield
>>
>> dsiembab01@gmail.com wrote:
>>> function check_memory_usage(&$memory)
>>> {
>>> $memory[] = memory_get_usage();
>>> return $memory;
>>> }
>>>
>>> something like this?
>>> you can put it wherever you like and returns an array for further
>>> processing. You could optionally add a second argument to set the
>>> index to a name and check if the name exists to add 1 to the end of
>>> the name so your indexes stay maintained.
>>>
>

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Memory investigation

you could read this
http://xdebug.org/docs/execution_trace
and then parse trace files to get the memory usage
create a global to store user functions, I think user function do not
populate get_defined_functions(); until the function is called;
$t = get_defined_functions();
$userFunctions = $t['user'];
$file = xdebug_get_tracefile_name();
parse the file and then create a function to call the said variable.
I think you can figure it out.
larry@garfieldtech.com wrote:
> Currently it's mostly procedural with some components that are OO. I
> suspect most of the memory sinks are large arrays (we have a lot of
> them), but I am not certain of that. Hence my interest in more accurate
> investigation tools.
>
> --Larry Garfield
>
> dsiembab01@gmail.com wrote:
>> couple questions Larry is this application composed of classes or
>> straight up no holes barred procedural code?
>>
>> larry@garfieldtech.com wrote:
>>> That's not really what I'm after. Let me try an example:
>>>
>>> function foo($id) {
>>> static $foos = array();
>>>
>>> if (empty($foos[$id]) {
>>> $foos[$id] = load_foo($id);
>>> }
>>> return $foos[$id];
>>> }
>>>
>>> When load_foo() is slow (e.g., lots of DB traffic or remote-server
>>> calls or whatever), such caching can have a significant performance
>>> boost. Sometime after foo() has been called 15 times from 30 places
>>> in code, when I get to the end of the request (or just every time I
>>> call foo() would be fine) I want to be able to do something like:
>>>
>>> $cost = get_memory_used_by($foos);
>>>
>>> So that I can determine how much memory that caching is costing me
>>> over the lifetime of the page, and determine if it's a worthwhile
>>> trade-off.
>>>
>>> --Larry Garfield
>>>
>>> dsiembab01@gmail.com wrote:
>>>> function check_memory_usage(&$memory)
>>>> {
>>>> $memory[] = memory_get_usage();
>>>> return $memory;
>>>> }
>>>>
>>>> something like this?
>>>> you can put it wherever you like and returns an array for further
>>>> processing. You could optionally add a second argument to set the
>>>> index to a name and check if the name exists to add 1 to the end of
>>>> the name so your indexes stay maintained.
>>>>
>>

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php