> 120 meg of json analyzed by the browser...

Hi..

I've built http://mediabeez.ws/htmlMicroscope/ (lgpl), which provides
a way to look at very big arrays in the browser.

I'm kinda stuck at a 120 - 200Mb data-size limit, and have exhausted
all my ideas on how to increase that limit further.
My reasoning is that i have a gigabyte free memory on my machine, and
want to use that to look at an array of at least 800mb.
I know, it's a totally academic feature at the moment.

But still, i would like feedback on how to increase the datasize limit
to actual-free-RAM.
If it's at all possible.
I would greatly appreciate feedback from webbrowser developers.

If you have some time and energy, please download the 1.3.0-beta on my
site, and have a look to see if you can find something to improve.

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: > 120 meg of json analyzed by the browser...

Rene Veerman wrote:
> Hi..
>
> I've built http://mediabeez.ws/htmlMicroscope/ (lgpl), which provides
> a way to look at very big arrays in the browser.
>
> I'm kinda stuck at a 120 - 200Mb data-size limit, and have exhausted
> all my ideas on how to increase that limit further.
> My reasoning is that i have a gigabyte free memory on my machine, and
> want to use that to look at an array of at least 800mb.
> I know, it's a totally academic feature at the moment.
>
> But still, i would like feedback on how to increase the datasize limit
> to actual-free-RAM.
> If it's at all possible.
> I would greatly appreciate feedback from webbrowser developers.
>
> If you have some time and energy, please download the 1.3.0-beta on my
> site, and have a look to see if you can find something to improve.

Use get_memory() to see just how much memory you're using. Somewhere in
your script you are probably storing much more memory than you think.

Cheers,
Rob.
--
http://www.interjinn.com
Application and Templating Framework for PHP

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: > 120 meg of json analyzed by the browser...

On Thu, Jan 28, 2010 at 4:04 PM, Robert Cummings wrote:
> Use get_memory() to see just how much memory you're using. Somewhere in your
> script you are probably storing much more memory than you think.

My functions do print the memory usage;
I'm just wondering why the array returned is 1/5th of the desired (and
allocated) size.


global $hmGenKeys;
$hmGenKeys = 0;
global $hmGenKeysDone;
$hmGenKeysDone = 0;
function htmlMicroscope_generateRandomArray ($maxKeys, $maxDepth,
$maxDuration=-1, $maxMem=-1) {
global $hmGenKeys;
global $hmGenKeysDone;

$r = array();
$l1 = false;
if ($maxKeys!==null) {
$hmGenKeys = $maxKeys;
$l1 = true;
}

$hmGenKeys--;
if ($hmGenKeys<=0) return false;
if ($maxDepth<=0) return false;
if ($l1) {
srand(htmlMicroscope_randMakeSeed());
while ($hmGenKeys > 0) {
$hmGenKeys--;
$hmGenKeysDone++;
file_put_contents('countdown.txt',
'k:'.number_format($hmGenKeysDone,0,'.',',').
' m:'.number_format(memory_get_usage(true),0,'.',',')
);
if ($maxMem!=-1 && memory_get_usage(true) > $maxMem) return $r;
if ($maxDuration!=-1 && $maxDuration < getDuration()) return $r;

switch (rand(1,2)) {
case 1 :
$next = htmlMicroscope_generateRandomArray (null, $maxDepth-1,
$maxDuration, $maxMem);
if ($next!==false)
$r += array(
htmlMicroscope_randomValue(4) => $next
);
break;
case 2 :
$r += array(
htmlMicroscope_randomValue(4) => htmlMicroscope_randomValue(20)
);
break;
}
}
} else {
$range = rand(0,50);
for ($i=0; $i<$range; $i++) {
$hmGenKeys--;
$hmGenKeysDone++;
file_put_contents('countdown.txt',
'k:'.number_format($hmGenKeysDone,0,'.',',').
' m:'.number_format(memory_get_usage(true),0,'.',',')
);
if ($maxMem!=-1 && memory_get_usage(true) > $maxMem) return $r;
if ($maxDuration!=-1 && $maxDuration < getDuration()) return $r;

switch (rand(1,2)) {
case 1 :
$next = htmlMicroscope_generateRandomArray (null, $maxDepth-1,
$maxDuration, $maxMem);
if ($next!==false)
$r += array(
htmlMicroscope_randomValue(4) => $next
);
break;
case 2 :
$r += array(
htmlMicroscope_randomValue(4) => htmlMicroscope_randomValue(20)
);
break;
}
}
}

if (($hmGenKeysDone/7919)==round($hmGenKeysDone/7919)) sleep(1);

return $r;
}

function htmlMicroscope_randomValue($maxLength) {
$r = '';
switch (rand(0,9)) {
case 0 : $r = rand (0,100000000); break;
case 1 : $r = rand (0,100000000) / rand(1,100) / 3; break;
default:
switch (rand(0,1)) {
case 0:
$rnd = rand(1,$maxDepth);
for ($i = 0; $i < $rnd; $i++) {
$r.= unichr(rand(0,hexdec('ffff')));
}
break;
case 1:
for ($i = 0; $i < $maxLength; $i++) {
$r.=chr(rand(ord('a'),ord('z')));;
}
break;
}
break;
}
//echo $r.'
'.$maxLength.'
';
return $r;
}

function htmlMicroscope_randMakeSeed() {
list($usec, $sec) = explode(' ', microtime());
return (float) $sec + ((float) $usec * 100000);
}

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: > 120 meg of json analyzed by the browser...

Oh, i forgot to mention that firefox takes about a gigabyte of memory
after having stalled at "200mb parsed" in a 330mb document..

And despite using setTimeout(), firefox frequently freezes (for about
2 to 10 minutes), before updating the decoding-status display again.

I'd really appreciate someone checking my non-eval() json
parser-decoder to see if my code is at fault, or if i've triggered a
firefox 'bug'.

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: > 120 meg of json analyzed by the browser...

Rene Veerman wrote:
> Oh, i forgot to mention that firefox takes about a gigabyte of memory
> after having stalled at "200mb parsed" in a 330mb document..
>
> And despite using setTimeout(), firefox frequently freezes (for about
> 2 to 10 minutes), before updating the decoding-status display again.
>
> I'd really appreciate someone checking my non-eval() json
> parser-decoder to see if my code is at fault, or if i've triggered a
> firefox 'bug'.

in all honesty I'm kinda wondering why anybody would want to look at
120+mb of array data in a web browser - do you have any use-cases?

as for the firefox side of things, it's built and designed to handle web
pages, documents, extensions and streaming media - when the rest of the
world is focussed on making things smaller, to optimise for making
things bigger normally doesn't come in to the equation.

regards though - challenges are what make us better.

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: > 120 meg of json analyzed by the browser...

Op 1/28/10 5:03 PM, Rene Veerman schreef:
> Oh, i forgot to mention that firefox takes about a gigabyte of memory
> after having stalled at "200mb parsed" in a 330mb document..
>
> And despite using setTimeout(), firefox frequently freezes (for about
> 2 to 10 minutes), before updating the decoding-status display again.
>
> I'd really appreciate someone checking my non-eval() json
> parser-decoder to see if my code is at fault, or if i've triggered a
> firefox 'bug'.

just guessing but I doubt you have a real issue in your parser-decoder,
the memory used by firefox seems reasonable to my untrained eye - I'd guess
that a factor 5 memory overhead is normal given the amount of abstraction
involved in the browser doing it's thing.

pretty cool what your trying to do - but, totally nuts of course :)

I would think that you're only recourse really is to chunk the output
of both the server and the browser so that you can, theoretically, page
through the data structure in the browser ... might be totally inpractical,
if not impossible. not to mention you likely to have to use file-based storage
for the chunks of output on the server side to avoid running out of mem.

hard problem!

>


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: > 120 meg of json analyzed by the browser...

--=-lzkMSL7tJriJyrhfEMJJ
Content-Type: text/plain
Content-Transfer-Encoding: 7bit

On Thu, 2010-01-28 at 17:30 +0100, Jochem Maas wrote:

> Op 1/28/10 5:03 PM, Rene Veerman schreef:
> > Oh, i forgot to mention that firefox takes about a gigabyte of memory
> > after having stalled at "200mb parsed" in a 330mb document..
> >
> > And despite using setTimeout(), firefox frequently freezes (for about
> > 2 to 10 minutes), before updating the decoding-status display again.
> >
> > I'd really appreciate someone checking my non-eval() json
> > parser-decoder to see if my code is at fault, or if i've triggered a
> > firefox 'bug'.
>
> just guessing but I doubt you have a real issue in your parser-decoder,
> the memory used by firefox seems reasonable to my untrained eye - I'd guess
> that a factor 5 memory overhead is normal given the amount of abstraction
> involved in the browser doing it's thing.
>
> pretty cool what your trying to do - but, totally nuts of course :)
>
> I would think that you're only recourse really is to chunk the output
> of both the server and the browser so that you can, theoretically, page
> through the data structure in the browser ... might be totally inpractical,
> if not impossible. not to mention you likely to have to use file-based storage
> for the chunks of output on the server side to avoid running out of mem.
>
> hard problem!
>
> >
>
>


You could page through the data and make it look like it's happening all
in the browser with a bit of clever ajax

Thanks,
Ash
http://www.ashleysheridan.co.uk



--=-lzkMSL7tJriJyrhfEMJJ--

Re: Re: > 120 meg of json analyzed by the browser...

At 200Mb/330Mb parsing, i have released 200Mb of html comment nodes,
and should have accumulated only 200Mb of javascript array/object.
it's _just_ the data, no HTML has been generated yet.
I accept a 5x overhead for turning it into HTML, but wonder why
firefox
a) stops updating the screen despite regular .setTimeout() pauses
b) accumulates 1 gigabyte of memory after only 200Mb has been processed.

And i dont think paging is possible, as i want to display things like the path
(of parent nodes) and sibling list for each sub-array in the data.


On Thu, Jan 28, 2010 at 5:30 PM, Jochem Maas wrote:
> just guessing but I doubt you have a real issue in your parser-decoder,
> the memory used by firefox seems reasonable to my untrained eye - I'd guess
> that a factor 5 memory overhead is normal given the amount of abstraction
> involved in the browser doing it's thing.
>
> pretty cool what your trying to do - but, totally nuts of course :)
>
> I would think that you're only recourse really is to chunk the output
> of both the server and the browser so that you can, theoretically, page
> through the data structure in the browser ... might be totally inpractical,
> if not impossible. not to mention you likely to have to use file-based storage
> for the chunks of output on the server side to avoid running out of mem.
>
> hard problem!
>
>>
>
>

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: > 120 meg of json analyzed by the browser...

--001517477fb0f2c3a2047e3c62e4
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Jan 28, 2010 at 5:32 PM, Ashley Sheridan
wrote:

>
> You could page through the data and make it look like it's happening all in
> the browser with a bit of clever ajax
>

Ok, good point.

Maybe JSON-transport > javascript parsing just has it's limit at just over
100 meg.

Accepting the fact that it would make the serverside code
much more complex and thus harder to port,
the data would imo need to be shoved into a db, because i doubt the
flat-file storage of $_SESSION would work well.

Anyone know of a piece of code that can pump a 1000-level
deep array of 2 gigabyte into a mysql db and
retrieve subtrees (startID, depth, filters?) efficiently?

--001517477fb0f2c3a2047e3c62e4--