memory doesn"t free after lexical block

Hello List!
I have a quick question about memory release in perl:

{
my @array;

foreach my $n (1..1e7 ) {
push @array, $n;
print "$n\n";
}
}

print "sleeping\n";
sleep 600;

after the code block, I epxect memory usage to drop to almost zero
because @array went out of scope. but when I do top after it executes
after the code block it still has huge memory usage..

Could anyone give me some explanation?

Thanks!
Jim


--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: memory doesn"t free after lexical block

On 5/18/11 Wed May 18, 2011 5:16 PM, "Jim Green"
scribbled:

> Hello List!
> I have a quick question about memory release in perl:
>
> {
> my @array;
>
> foreach my $n (1..1e7 ) {
> push @array, $n;
> print "$n\n";
> }
> }
>
> print "sleeping\n";
> sleep 600;
>
> after the code block, I epxect memory usage to drop to almost zero
> because @array went out of scope. but when I do top after it executes
> after the code block it still has huge memory usage..
>
> Could anyone give me some explanation?

See the explanation in 'perldoc -q array'

"How can I free an array or hash so my program shrinks?"

Perl hangs on to the memory and does not return it back to the operating
system.



--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: memory doesn"t free after lexical block

On 11-05-18 08:25 PM, Jim Gibson wrote:
> Perl hangs on to the memory and does not return it back to the operating
> system.

Under UNIX, et al. all processes hang on to their memory because it's
too time consuming to give it back.


--
Just my 0.00000002 million dollars worth,
Shawn

Confusion is the first step of understanding.

Programming is as much about organization and communication
as it is about coding.

The secret to great software: Fail early & often.

Eliminate software piracy: use only FLOSS.

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: memory doesn"t free after lexical block

On Wed, May 18, 2011 at 05:16:23PM -0700, Jim Green wrote:
> Hello List!
> I have a quick question about memory release in perl:
>
> {
> my @array;
>
> foreach my $n (1..1e7 ) {
> push @array, $n;
> print "$n\n";
> }
> }
>
> print "sleeping\n";
> sleep 600;
>
> after the code block, I epxect memory usage to drop to almost zero
> because @array went out of scope. but when I do top after it executes
> after the code block it still has huge memory usage..
>
> Could anyone give me some explanation?

In general perl won't release memory back to the system, but will keep
it around to be (possibly) reused later. If you subsequently create
another similarly sized array you should notice that memory use remains
roughly the same.

--
Paul Johnson - paul@pjcj.net
http://www.pjcj.net

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: memory doesn"t free after lexical block

On 18 May 2011 20:30, Paul Johnson wrote:
> On Wed, May 18, 2011 at 05:16:23PM -0700, Jim Green wrote:
>> Hello List!
>> I have a quick question about memory release in perl:
>>
>> {
>> =A0 =A0 my @array;
>>
>> =A0 =A0 foreach my $n (1..1e7 ) {
>> =A0 =A0 =A0 =A0 push @array, $n;
>> =A0 =A0 =A0 =A0 print "$n\n";
>> =A0 =A0 }
>> }
>>
>> print "sleeping\n";
>> sleep 600;
>>
>> after the code block, I epxect memory usage to drop to almost zero
>> because @array went out of scope. but when I do top after it executes
>> after the code block it still has huge memory usage..
>>
>> Could anyone give me some explanation?
>
> In general perl won't release memory back to the system, but will keep
> it around to be (possibly) reused later. =A0If you subsequently create
> another similarly sized array you should notice that memory use remains
> roughly the same.

lets say I have a really big perl program, if I let the it run, the
memory usage will always increase until it exits?

is this the same for other language like c++ or java?

Thank you all!
Jim

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: memory doesn"t free after lexical block

On 11-05-18 08:36 PM, Jim Green wrote:
> is this the same for other language like c++ or java?

For all processes. That's why deamons periodically respawn; to clean up
messes like this.


--
Just my 0.00000002 million dollars worth,
Shawn

Confusion is the first step of understanding.

Programming is as much about organization and communication
as it is about coding.

The secret to great software: Fail early & often.

Eliminate software piracy: use only FLOSS.

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: memory doesn"t free after lexical block

On 18 May 2011 20:39, Shawn H Corey wrote:
> On 11-05-18 08:36 PM, Jim Green wrote:
>>
>> is this the same for other language like c++ or java?
>
> For all processes. =A0That's why deamons periodically respawn; to clean u=
p
> messes like this.

Thank you! I just wish this is not a deficiency in Perl language itself.
If it is the same for other language, I am happy!

Jim

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: memory doesn"t free after lexical block

>>>>> "SHC" == Shawn H Corey writes:

SHC> On 11-05-18 08:25 PM, Jim Gibson wrote:
>> Perl hangs on to the memory and does not return it back to the operating
>> system.

SHC> Under UNIX, et al. all processes hang on to their memory because it's
SHC> too time consuming to give it back.

that isn't the case. the brk() system call can return memory to the OS
but it requires the free memory to be at the end of the current memory
space. that is difficult to do with standard malloc/free calls. there
are ways to manage ram in a process so it can be returned to the OS. it
isn't a time issue but an design and memory management issue.

uri

--
Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com --
----- Perl Code Review , Architecture, Development, Training, Support ------
--------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: memory doesn"t free after lexical block

>>>>> "SHC" == Shawn H Corey writes:

SHC> On 11-05-18 08:36 PM, Jim Green wrote:
>> is this the same for other language like c++ or java?

SHC> For all processes. That's why deamons periodically respawn; to clean
SHC> up messes like this.

again, somewhat incorrect. a properly well written daemon will not leak
ram and can run for a long time. note that many daemons in the OS don't
respawn at all. they have been debugged and are long running without
leaks. perl itself will generally not leak but it is easy to create perl
data structure which don't get properly freed up (reference loops) and
that can leak. there have been reports of perl itself leaking in various
versions under certain circumstances but the average perl program won't
hit those. any leaks then are usually the fault of the perl coder, not
perl.

uri

--
Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com --
----- Perl Code Review , Architecture, Development, Training, Support ------
--------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: memory doesn"t free after lexical block

>>>>> "JG" == Jim Green writes:

JG> On 18 May 2011 20:39, Shawn H Corey wrote:
>> On 11-05-18 08:36 PM, Jim Green wrote:
>>>=20
>>> is this the same for other language like c++ or java?
>>=20
>> For all processes. =A0That's why deamons periodically respawn; to clea=
n up
>> messes like this.

JG> Thank you! I just wish this is not a deficiency in Perl language itse=
lf.
JG> If it is the same for other language, I am happy!

there are many ways to leak and waste ram. some langs have garbage
collection which is safer than perl's reference counting for ram
cleanup. but almost all serious leaks are the fault of the coder. and
you can waste ram in any language just with poor coding. nothing can
stop a bad coder from building up ginormous data structures and never
freeing them and always adding more to them. seen it many times. just
poor designs and bad code. perl is no more or less susceptible to that
issue.

uri

--=20
Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com =
--
----- Perl Code Review , Architecture, Development, Training, Support ----=
--
--------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com -------=
--

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: memory doesn"t free after lexical block

--90e6ba6e86783a8afc04a39a6ed3
Content-Type: text/plain; charset=ISO-8859-1

Uri, how would you change the original code to get it to free up memory?
I've run into this issue myself quite a few times. I've also seen perl
consume an entire cpu core without even trying. Could you rework the
original scriptlet to show how you would free the memory at the end or
direct us to a resource that explains this behavior so that we can have more
efficient scripts?

Often in the business world admins will see the resources that perl is using
and balk saying that it is taking up too many resources. I'd like to be
able to minimize the footprint of my scripts.

On Wed, May 18, 2011 at 11:22 PM, Uri Guttman wrote:

> >>>>> "JG" == Jim Green writes:
>
> JG> On 18 May 2011 20:39, Shawn H Corey wrote:
> >> On 11-05-18 08:36 PM, Jim Green wrote:
> >>>
> >>> is this the same for other language like c++ or java?
> >>
> >> For all processes. That's why deamons periodically respawn; to clean
> up
> >> messes like this.
>
> JG> Thank you! I just wish this is not a deficiency in Perl language
> itself.
> JG> If it is the same for other language, I am happy!
>
> there are many ways to leak and waste ram. some langs have garbage
> collection which is safer than perl's reference counting for ram
> cleanup. but almost all serious leaks are the fault of the coder. and
> you can waste ram in any language just with poor coding. nothing can
> stop a bad coder from building up ginormous data structures and never
> freeing them and always adding more to them. seen it many times. just
> poor designs and bad code. perl is no more or less susceptible to that
> issue.
>
> uri
>
> --
> Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com--
> ----- Perl Code Review , Architecture, Development, Training, Support
> ------
> --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com---------
>
> --
> To unsubscribe, e-mail: beginners-unsubscribe@perl.org
> For additional commands, e-mail: beginners-help@perl.org
> http://learn.perl.org/
>
>
>

--90e6ba6e86783a8afc04a39a6ed3--

Re: memory doesn"t free after lexical block

>>>>> "SR" == Sheppy R writes:

SR> Uri, how would you change the original code to get it to free up
SR> memory? I've run into this issue myself quite a few times. I've
SR> also seen perl consume an entire cpu core without even trying.
SR> Could you rework the original scriptlet to show how you would free
SR> the memory at the end or direct us to a resource that explains
SR> this behavior so that we can have more efficient scripts?

perl cannot return ram to the OS. i said processes when do very special
memory management can do this but it is very rare. don't expect to even
try this as you can't from perl.

if you design your perl code well you shouldn't have memory leaks which
is the usual way to lose ram. other ways are just bad code that builds
up and up and never frees anything. you free stuff in perl indirectly
when the reference count hits zero.

freeing ram is just natural in perl if you declare things lexically in a
sub or block. when you leave the sub/block they will be freed back to
perl (not to the OS).

also consuming a cpu core is a cpu issue not a ram issue. it means
either your program is cpu bound (which is very possible) or you have an
infinite loop.

SR> Often in the business world admins will see the resources that
SR> perl is using and balk saying that it is taking up too many
SR> resources. I'd like to be able to minimize the footprint of my
SR> scripts.

i can't help from a distance and without code. if you are serious, i can
consult with your place and help out that way. fixing ram issues is
usually not trivial nor something beginners know how to do. this is
beyond the scope of a beginner's list and i suggest you get professional
assistance (me in particular :).

uri

--
Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com --
----- Perl Code Review , Architecture, Development, Training, Support ------
--------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: memory doesn"t free after lexical block

>>>>> "JG" == Jim Green writes:

JG> I have a quick question about memory release in perl:

JG> {
JG> my @array;

JG> foreach my $n (1..1e7 ) {
JG> push @array, $n;
JG> print "$n\n";
JG> }
JG> }

someone asked about the original code. this is a silly example in
several ways. it is building up an array with just the indexes as values
(actually off by one but that doesn't matter here). so it it not worth
fixing as it is just an example of large ram usage for no reason.

as i said in another post, @array will be freed up at the block exit but
it won't be freed back to the OS. the process will own that ram until it
exits. but perl will reuse that ram before it allocates more which is
usually fine for most programs.

there is no direct way to reduce the ram usage in that code. it is just
designed to suck up ram just to see it being done. you couldn't free
that up in most any lang as none have a way to free it back to the
OS. you can do that in c IF and only IF you don't call malloc/free and
you call brk/sbrk yourself and manage your own ram. it is extremely
difficult for a lang to do that for you as allocation and freeing of ram
will be random and not controlled so you can't free the ram to the
OS. you can only free the top contiguous ram to the OS with a brk
call. this gets into seriously tricky stuff that i won't get into more
here as it won't affect the perl code.

if you have a more realistic example with large ram usage, that possibly
could be addressed. even so, this is not a typical beginner topic.

uri

--
Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com --
----- Perl Code Review , Architecture, Development, Training, Support ------
--------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: memory doesn"t free after lexical block

On 11-05-19 01:39 AM, Sheppy R wrote:
> how would you change the original code to get it to free up memory?

Put the processing in a sub-process. When it dies, the memory is
returned to the system.

See:
perldoc -f fork
perldoc perlipc


--
Just my 0.00000002 million dollars worth,
Shawn

Confusion is the first step of understanding.

Programming is as much about organization and communication
as it is about coding.

The secret to great software: Fail early & often.

Eliminate software piracy: use only FLOSS.

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: memory doesn"t free after lexical block

What is being told in here, is that:


foreach my $m (1..1e7 ) {
my @array;

foreach my $n (1..1e7 ) {
push @array, $n;
}

print "in\n";
}

print "sleeping\n";
sleep 600;

Stops growing in memory.

In normal execution, when the memory is no longer used, the OS will cache i=
t.

Best Regards
Marcos

On Thu, May 19, 2011 at 14:26, Shawn H Corey wrote:
> On 11-05-19 01:39 AM, Sheppy R wrote:
>>
>> how would you change the original code to get it to free up memory?
>
> Put the processing in a sub-process. =A0When it dies, the memory is retur=
ned
> to the system.
>
> See:
> perldoc -f fork
> perldoc perlipc
>
>
> --
> Just my 0.00000002 million dollars worth,
> =A0Shawn
>
> Confusion is the first step of understanding.
>
> Programming is as much about organization and communication
> as it is about coding.
>
> The secret to great software: =A0Fail early & often.
>
> Eliminate software piracy: =A0use only FLOSS.
>
> --
> To unsubscribe, e-mail: beginners-unsubscribe@perl.org
> For additional commands, e-mail: beginners-help@perl.org
> http://learn.perl.org/
>
>
>



--=20
Marcos Rebelo
http://www.oleber.com/
Milan Perl Mongers leader https://sites.google.com/site/milanperlmongers/
Webmaster of http://perl5notebook.oleber.com

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/