Re: out of memory<04e40e32-58da-4a64-9a34-6d334135ca55@a39g2000pre.googlegroups.com><p58125

On 2007-11-28 22:54, Ben Morrow wrote:
> Quoth aniait :
>> Is there any way to solve the 'out of memory exception' of perl on
>> windows? I got this when i tried to dump a large hash into a file
>> using Dumper.
>
> Increase the size of your swapfile :). Windows machines usually don't
> have anywhere near enough swap.

OTOH perl hashes tend to have absolutely abysmal performance as soon as
swapping is required, so increasing the size of RAM is probably less
painful (although a bit more expensive).

But the main problem with Dumper is that it constructs one huge string,
so the problem can be solved by walking the hash and writing each
element to disk immediately. Storable and YAML at least offer functions
to read from/write to files, although I don't know offhand if they
really stream or if they need a huge buffer.

hp


--
_ | Peter J. Holzer | It took a genius to create [TeX],
|_|_) | Sysadmin WSR | and it takes a genius to maintain it.
| | | hjp@hjp.at | That's not engineering, that's art.
__/ | http://www.hjp.at/ | -- David Kastrup in comp.text.tex

Re: out of memory<04e40e32-58da-4a64-9a34-6d334135ca55@a39g2000pre.googlegroups.com><p58125

Quoth "Peter J. Holzer" :
> On 2007-11-28 22:54, Ben Morrow wrote:
> > Quoth aniait :
> >> Is there any way to solve the 'out of memory exception' of perl on
> >> windows? I got this when i tried to dump a large hash into a file
> >> using Dumper.
> >
> > Increase the size of your swapfile :). Windows machines usually don't
> > have anywhere near enough swap.
>
> OTOH perl hashes tend to have absolutely abysmal performance as soon as
> swapping is required, so increasing the size of RAM is probably less
> painful (although a bit more expensive).

OTGH I was assuming that perl wasn't in fact filling physical memory,
and there were other things running that could usefully be swapped out.
Windows is rather funny about swap usage: for instance, IIRC all shared
libraries are copied into swap and demand-paged from there, rather than
from the original file.

> But the main problem with Dumper is that it constructs one huge string,
> so the problem can be solved by walking the hash and writing each
> element to disk immediately.

Yes. I've been thinking lately that a generic 'walk this data structure'
module, sort-of like File::Find, would be handy. Getting it right is
*far* from trivial, especially if you want to pick up 'sideways' stuff
like magic objects and closed-over lexicals, and a lot of modules seem
to have their own version. Maybe I'll generalise the code in
Clone::Closure slightly, and nick some bits from Storable...

> Storable and YAML at least offer functions
> to read from/write to files, although I don't know offhand if they
> really stream or if they need a huge buffer.

Storable at least really streams.

Ben