How to avoid using the slow array subscripting operator?

--20cf307f3a3670fa2e04a426c378
Content-Type: text/plain; charset=ISO-8859-1

Hi all:
I translated a program from c to perl.but the perl program cost 15 seconds
compare to the original c one cost only less than 1 second, I guess this
might be the result of I literally translated the program, using a lot of
array subscripts. After I profile the perl program, it turned out my guess
was true.
for($i = 0; $i < @lines; $i++)
{
$sum += $lines[$_]->{foo_value} for ($i - $n + 1 .. $i);
}
sorry I can't post the whole code, but the lines above is the big framework,
I also omit all the boundary check. "$sum += $lines[$_]->{foo_value} for ($i
- $n2 + 1 .. $i)" this line consumes all the time.
I got a array, for each element of the array I have to sum up the $n
elements prior to the current one. I know I should use pop push shift
unshift all the perl list operators instead of subscripting, but I just
don't know how, would somebody tell me what is the perl way of doing this?

--20cf307f3a3670fa2e04a426c378--

Re: How to avoid using the slow array subscripting operator?

>>>>> "XL" == Xi Liu writes:

XL> I translated a program from c to perl.but the perl program cost 15 seconds
XL> compare to the original c one cost only less than 1 second, I guess this
XL> might be the result of I literally translated the program, using a lot of
XL> array subscripts. After I profile the perl program, it turned out my guess
XL> was true.

you need to learn real perl first and not code as in c. this is true for
most langs. each has idioms that work best and literal translations from
one to another are almost always bad.


XL> for($i = 0; $i < @lines; $i++)

first rule: don't use c style index loops when you can avoid them. that
line is much faster and more perlish as:

foreach my $i ( 0 .. $#lines )

XL> $sum += $lines[$_]->{foo_value} for ($i - $n + 1 .. $i);

XL> sorry I can't post the whole code, but the lines above is the big
XL> framework, I also omit all the boundary check. "$sum +=
XL> $lines[$_]->{foo_value} for ($i - $n2 + 1 .. $i)" this line
XL> consumes all the time.

XL> I got a array, for each element of the array I have to sum up the
XL> $n elements prior to the current one. I know I should use pop push
XL> shift unshift all the perl list operators instead of subscripting,
XL> but I just don't know how, would somebody tell me what is the perl
XL> way of doing this?

that doesn't seem to be what the code is doing. $sum isn't just a sum,
it is a sum of sums as you have a two dimensional loop. did you
translate the code incorrectly or can you more clearly specify what is
being summed here?

you can get speedups in other ways too. you can do a map or slice of the
array and call sum (in List::Util) on that list. it will be much faster
than the += loop you have there. also you are accessing {foo_value} over
and over for each line (since the loop is 2 levels deep). a better
algorithm would do a sum of sums where you sum one set and then add that
to the next set. i won't get into that since the second loop logic isn't
clear to me (which set of elements are being summed and resummed). that
is more your problem. but this can be sped up a great deal with better
coding.

uri

--
Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com --
----- Perl Code Review , Architecture, Development, Training, Support ------
--------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: How to avoid using the slow array subscripting operator?

--20cf3033451dc2fd8004a428a3cb
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks for your help!
I am sorry I missed something important in the code snippet.
for($i =3D 0; $i < @lines; $i++)
{
$sum =3D 0;
$sum +=3D $lines[$_]->{foo_value} for ($i - $n + 1 .. $i);
push @res, $sum;
}
this is what I intend.I try to make it clear and removed something so that
change the original intend without noticing it, sorry.
I write a little program:

#!/usr/bin/perl
use warnings;
use List::Util qw(sum);
push @lines, {foo_value =3D> 6000 + int rand(4000)} for 0 .. 60000;
my $n =3D 100;
for my $i (99 .. 60000)
{
my ($sum1, $sum2, $sum3);
$sum1 +=3D $lines[$_]->{foo_value} for ($i - $n + 1 .. $i);
map {$sum2 +=3D $_->{foo_value}} @lines[$i - $n + 1 .. $i];
$sum3 =3D sum (map {$_->{foo_value}} @lines[$i - $n + 1 .. $i]);
}

and profile it, it turns out the result
3.00s $sum1 +=3D $lines[$_]->{foo_value} for ($i - $n + 1 .. $i);
1.71s map {$sum2 +=3D $_->{foo_value}} @lines[$i - $n + 1 .. $i];
2.10s $sum3 =3D sum (map {$_->{foo_value}} @lines[$i - $n + 1 .. $i]);

using map on sliced list makes it better but still not acceptable.Any more
ideas?

and I used to think the "foreach $i (0..$#lines)" and "for($i =3D 0; $i <
@lines; $i++)" are the same.
but the prior cost 66=B5s , a great victory compared to the second's 50ms.
how should I suppose to know they have efficiency differences? *:-)*
Thanks for your help again!

--20cf3033451dc2fd8004a428a3cb--

Re: How to avoid using the slow array subscripting operator?

>>>>> "XL" == Xi Liu writes:

XL> Thanks for your help!
XL> I am sorry I missed something important in the code snippet.
XL> for($i =3D 0; $i < @lines; $i++)
XL> {
XL> $sum =3D 0;
XL> $sum +=3D $lines[$_]->{foo_value} for ($i - $n + 1 .. $i);
XL> push @res, $sum;
XL> }
XL> this is what I intend.I try to make it clear and removed something so=
that
XL> change the original intend without noticing it, sorry.

a good rule to all newbies here. post all the relevent code. removing
lines can obscure your real intent as this did here.


XL> I write a little program:

XL> #!/usr/bin/perl
XL> use warnings;
XL> use List::Util qw(sum);

the best way to time code is with the Benchmark module.=20

XL> push @lines, {foo_value =3D> 6000 + int rand(4000)} for 0 .. 60000;
XL> my $n =3D 100;
XL> for my $i (99 .. 60000)
XL> {
XL> my ($sum1, $sum2, $sum3);
XL> $sum1 +=3D $lines[$_]->{foo_value} for ($i - $n + 1 .. $i);
XL> map {$sum2 +=3D $_->{foo_value}} @lines[$i - $n + 1 .. $i];
XL> $sum3 =3D sum (map {$_->{foo_value}} @lines[$i - $n + 1 .. $i]);
XL> }

none of those do the push you mentioned. and if they did, they could use
the saved sums to shorten the later summing loops as they sum
overlapping values. that is the key to saving time here.

XL> and profile it, it turns out the result
XL> 3.00s $sum1 +=3D $lines[$_]->{foo_value} for ($i - $n + 1 .. $i);
XL> 1.71s map {$sum2 +=3D $_->{foo_value}} @lines[$i - $n + 1 .. $i];

that map is not cool. it isn't using the resulting list of the map. even
though it is the fastest one here, i wouldn't use it.

XL> 2.10s $sum3 =3D sum (map {$_->{foo_value}} @lines[$i - $n + 1 .. $i]);

XL> using map on sliced list makes it better but still not acceptable.Any=
more
XL> ideas?

what i said above. you are doing the same work multiple times by not
keeping the earlier sums and using them.

XL> and I used to think the "foreach $i (0..$#lines)" and "for($i =3D 0; =
$i <
XL> @lines; $i++)" are the same.
XL> but the prior cost 66=B5s , a great victory compared to the second's =
50ms.
XL> how should I suppose to know they have efficiency differences? *:-)*

it is fairly obvious on the surface. the .. just builds a list and
internally gets the next one. the c style ++ loop has to execute several
perl operations for each iteration. a rule of thumb in perl speed (not
always true but most often it is) is the more perl code the slower. the
more you stay inside perl's guts (in c) the faster. the perl operator
loop is slow so use fewer perl ops. the c style loop uses perl ops. the
foreach loop doesn't.

another idea is to loop over the lines themselves and not their
indexes. again, your math eludes me and i won't get into it. you seem to
be summing the 100 lines at each possible point in the main array. so
that means you are just summing he first and last 100 lines in a
triangle way and ALL the other lines are just 60,000 x 100 times added
in to the sum. so you could short circuit the whole middle with a single
loop and sum them up and then multiply that by 59,900 and get the same
value. that will blow your time away since you removed all the double
and slow looping stuff. again this is from a very cursory analysis of
the math which still makes NO sense to me. why are you summing
overlapping segments of the main array? if you know you are going to do
that, factor out the parts that i said and it will speed up massively.

rule: a better algorithm usually beats better code.

uri

--=20
Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com =
--
----- Perl Code Review , Architecture, Development, Training, Support ----=
--
--------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com -------=
--

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: How to avoid using the slow array subscripting operator?

--20cf3071cd0cafa61604a42aa929
Content-Type: text/plain; charset=ISO-8859-1

I know I am doing the repetitive and useless summing again and again. What
confuses me is that using the same algorithm, I mean

for my $i (99 .. 60000)
{
my $sum;
map {$sum += $_->{foo_value}} @lines[$i - $n + 1 .. $i];
push @res, $sum;
}

in perl and

for(int i =99; i <= 60000; i++)
{
int sum = 0;
for(int j = i - n +1; j < i; j++)
sum += lines[$j].foo_value;
res[i] = sum;
}

in c, there is a huge difference in efficiency.
the c program, even using the same stupid algorithm, the cost of time is
acceptable. So I translated it to perl , and avoid the slow subscripting
operations using map on sliced array, I suppose the perl program would be
slower, but I don't predict such a huge difference, you don't even need a
benchmark or profiling tool to notice the efficiency difference. This is
what I am confused.
Thanks for your patience!

--20cf3071cd0cafa61604a42aa929--

Re: How to avoid using the slow array subscripting operator?

>>>>> "XL" == Xi Liu writes:

XL> I know I am doing the repetitive and useless summing again and again.=
What
XL> confuses me is that using the same algorithm, I mean

XL> =A0for my $i (99 .. 60000)
XL> {
XL> =A0 =A0 =A0 my $sum;
XL> =A0 =A0 =A0 map {$sum +=3D $_->{foo_value}} @lines[$i - $n + 1 .. $i];
XL> =A0 =A0 =A0 push @res, $sum;
XL> }

XL> in perl and=A0

XL> for(int i =3D99; i <=3D 60000; i++)
XL> {
XL> =A0 =A0 =A0 int sum =3D 0;
XL> =A0 =A0 =A0 for(int j =3D i - n +1; j < i; =A0j++)
XL> =A0 =A0 =A0 =A0 =A0 sum +=3D lines[$j].foo_value;
XL> =A0 =A0 =A0 res[i] =3D sum;
XL> }

XL> in c, there is a huge difference in efficiency. the c program,
XL> =A0even using the same stupid algorithm, the cost of time is
XL> acceptable. So I translated it to perl , and avoid the slow
XL> subscripting operations using map on sliced array, I suppose the
XL> perl program would be slower, but I don't predict such a huge
XL> difference, you don't even need a benchmark or profiling tool to
XL> notice the=A0efficiency difference.=A0This is what

nothing to be confused about. this is the difference between a compiled
and interpreted language. the c code you have is compiled down to very
efficient machine code. a structure access in c is very cheap (just an
addition or so in machine code). a hash access in perl is much more
expensive. general perl code will never be close to the speed of
compiled code. specialized perl code using its guts like the regex
engine can be close to c code especially badly written code. the win for
perl is the much faster time coding up the program and the flexibility
of perl over c. i did over 20 years of c and spent half my time doing
stuff that perl does for me like memory management, manipulating data
structures, dynamic data issues, etc. my time is more valuable than the
computer's so i choose perl over c. this difference is well known by
anyone who deals with c and perl. it is why you can write c code inside
perl (with XS, Inline::C, etc). many cpan modules have c code in them
for speedup. many existing libraries are all in c and perl modules
provide wrappers for them. c is not going away and neither is perl. if
you want blazing speed on simple stuff like your code, keep it in
c. perl will never be fast enough for that.

uri

--=20
Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com =
--
----- Perl Code Review , Architecture, Development, Training, Support ----=
--
--------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com -------=
--

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: How to avoid using the slow array subscripting operator?

--bcaec5485cfc15d1e204a436d067
Content-Type: text/plain; charset=ISO-8859-1

OK, I think I got the idea.
I am going to change the useless code caching the result I have already
summed or inline c code to achieve a better performance. Really learned
something. Thank you!

--bcaec5485cfc15d1e204a436d067--

Re: How to avoid using the slow array subscripting operator?

On May 25, 10:05=A0pm, jason.li...@gmail.com (Xi Liu) wrote:
> Hi all:
> I translated a program from c to perl.but the perl program cost 15 second=
s
> compare to the original c one cost only less than 1 second, I guess this
> might be the result of I literally translated the program, using a lot of
> array subscripts. After I profile the perl program, it turned out my gues=
s
> was true.
> for($i =3D 0; $i < @lines; $i++)
> {
> =A0 =A0 $sum +=3D $lines[$_]->{foo_value} for ($i - $n + 1 .. $i);}
>
> sorry I can't post the whole code, but the lines above is the big framewo=
rk,
> I also omit all the boundary check. "$sum +=3D $lines[$_]->{foo_value} fo=
r ($i
> - $n2 + 1 .. $i)" this line consumes all the time.
> I got a array, for each element of the array I have to sum up the $n
> elements prior to the current one. I know I should use pop push shift
> unshift all the perl list operators instead of subscripting, but I just
> don't know how, would somebody tell me what is the perl way of doing this=
?


Another option, if Perl performance remains too slow after
your improvements, would be an XS extension interface
between Perl and C code.
See: perldoc perlxs

or, for a different approach and a much easier learning curve,
See: Inline::C and/or Inline::C-Cookbook.

--
Charles DeRykus


--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: How to avoid using the slow array subscripting operator?

On May 26, 5:44=A0am, jason.li...@gmail.com (Xi Liu) wrote:
> I know I am doing the repetitive and useless summing again and again. Wha=
t
> confuses me is that using the same algorithm, I mean
>
> =A0for my $i (99 .. 60000)
> {
> =A0 =A0 =A0 my $sum;
> =A0 =A0 =A0 map {$sum +=3D $_->{foo_value}} @lines[$i - $n + 1 .. $i];
> =A0 =A0 =A0 push @res, $sum;
>
> }
>
> in perl and
>
> for(int i =3D99; i <=3D 60000; i++)
> {
> =A0 =A0 =A0 int sum =3D 0;
> =A0 =A0 =A0 for(int j =3D i - n +1; j < i; =A0j++)
> =A0 =A0 =A0 =A0 =A0 sum +=3D lines[$j].foo_value;
> =A0 =A0 =A0 res[i] =3D sum;
>
> }
>
> in c, there is a huge difference in efficiency.
> the c program, =A0even using the same stupid algorithm, the cost of time =
is
> acceptable. So I translated it to perl , and avoid the slow subscripting
> operations using map on sliced array, I suppose the perl program would be
> slower, but I don't predict such a huge difference, you don't even need a
> benchmark or profiling tool to notice the efficiency difference. This is
> what I am confused.
> Thanks for your patience!

I wrote and tested a version of your program that uses caching and
compared it to the non cached example you provided. The cached version
runs in about 3% of the time that the non cached version does. So, as
Uri observed, a better choice of algorithm can improve things quite a
bit. (Note, the cached version could also be implemented in the C
version.)

My run produced this output (timed using the Time::HiRes module).

C:\Old_Data\perlp>perl t.pl
cached: 0.0327188968658447 sums: 59902
non_cached: 1.18266701698303 sums: 59902


The program is:

#!/usr/bin/perl
use strict;
use warnings;
use 5.012;
use Math::Random::MT::Auto qw/ rand srand /; # or just use the built
in 'rand'
use Time::HiRes qw/ time /; # I just
like using a 'better' rand!
use List::Util qw/ sum /;
use Data::Dumper;

srand(1); # to get the same random numbers from run to run for
consistancy in timing
my @lines;
for (0 .. 60000) {
push @lines, {foo_value =3D> int(rand 100000)};
}

my $n =3D 100;
my ($start, $end) =3D (99, 60000);

my $begin =3D time;
my $sum =3D sum map $_->{foo_value}, @lines[ $start - $n + 1 ..
$start ];

my @res =3D $sum;

for my $upper ($start+1 .. $end) {
$sum +=3D $lines[$upper]{foo_value} - $lines[$upper - $n]{foo_value};
push @res, $sum;
}
my $finish =3D time;

say "cached: ", $finish - $begin, " sums: " . @res;
#open FH, '>', 'cache.txt'; print FH Dumper \@res; close FH;
@res =3D ();

$begin =3D time;
for my $i (99 .. 60000) {
push @res, sum map {$_->{foo_value}} @lines[$i - $n + 1 .. $i];
}
$finish =3D time;

say "non_cached: ", $finish - $begin, " sums: " . @res;
#open FH, '>', 'non_cache.txt'; print FH Dumper \@res; close FH;


Chris


--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: How to avoid using the slow array subscripting operator?

On 2011-05-27 02:27, charley@pulsenet.com wrote:

> for (0 .. 60000) {

Be aware that this sets up a memory structure. Still 2 times faster than
looping:


$ time perl -wle '$i=0; 1 while $i++ < 1e4'
real 0m0.005s

$ time perl -wle '$i=0; 1 while $i++ < 1e5'
real 0m0.013s

$ time perl -wle '$i=0; 1 while $i++ < 1e6'
real 0m0.081s

$ time perl -wle '$i=0; 1 while $i++ < 1e7'
real 0m0.762s

$ time perl -wle '$i=0; 1 while $i++ < 1e8'
real 0m7.723s



$ time perl -wle '1 for 1..1e4'
real 0m0.005s

$ time perl -wle '1 for 1..1e5'
real 0m0.009s

$ time perl -wle '1 for 1..1e6'
real 0m0.047s

$ time perl -wle '1 for 1..1e7'
real 0m0.405s

$ time perl -wle '1 for 1..1e8'
real 0m3.997s


--
Ruud

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: How to avoid using the slow array subscripting operator?

On 26/05/2011 06:18, Uri Guttman wrote:
>>>>>> "XL" == Xi Liu writes:
>
> XL> I translated a program from c to perl.but the perl program cost 15 seconds
> XL> compare to the original c one cost only less than 1 second, I guess this
> XL> might be the result of I literally translated the program, using a lot of
> XL> array subscripts. After I profile the perl program, it turned out my guess
> XL> was true.
>
> you need to learn real perl first and not code as in c. this is true for
> most langs. each has idioms that work best and literal translations from
> one to another are almost always bad.
>
>
> XL> for($i = 0; $i< @lines; $i++)
>
> first rule: don't use c style index loops when you can avoid them. that
> line is much faster and more perlish as:
>
> foreach my $i ( 0 .. $#lines )

It is also rare to need access to the array index (even if you think you
do!). Perl is much more in its element if you loop over the elements of
an array instead of its indices:

foreach my $elem (@lines) {
:
}

Later on you publish this code, which uses a slice in this way

> for my $i (99 .. 60000)
> {
> my $sum;
> map {$sum += $_->{foo_value}} @lines[$i - $n + 1 .. $i];
> push @res, $sum;
> }

First of all, never use map in this way. It is a uniform list operator,
designed to apply the same transform to every element of its object list
and return a new list as a return value. Here you have map unnecessarily
building the new list and then then immediately discarding it. Your code
should be

$sum += $_->{foo_value}} foreach @lines[$i - $n + 1 .. $i];

or, to demonstrate my point about map

$sum += $_ foreach map $_->{foo_value}, @lines[$i - $n + 1 .. $i];

This last line maps the elements in the slice of @lines to a list of
their {foo_value} structure elements, and accumulates those into $sum.

I am sure that you could get your C program working with acceptable
speed, but you would need to write in Perl idioms rather than simply
transliterate the C code.

I would warn against writing 'C in Perl' together with embedded C, as it
makes the program far less maintainable. As you have not described the
purpose of your work, I guess that improved maintainability or
cross-platform functionality are amongst your goals?

Please persevere. Perl can be very fast, I assure you.

Rob


--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: How to avoid using the slow array subscripting operator?

>>>>> "RD" == Rob Dixon writes:

RD> On 26/05/2011 06:18, Uri Guttman wrote:

XL> for($i = 0; $i< @lines; $i++)
>>
>> first rule: don't use c style index loops when you can avoid them. that
>> line is much faster and more perlish as:
>>
>> foreach my $i ( 0 .. $#lines )

RD> It is also rare to need access to the array index (even if you think you
RD> do!). Perl is much more in its element if you loop over the elements of
RD> an array instead of its indices:

RD> foreach my $elem (@lines) {

i would normally have said that but he does need the index to setup the
inner loop index. as i said, the algorithm is all screwy with massive
duplicate summings.

uri

--
Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com --
----- Perl Code Review , Architecture, Development, Training, Support ------
--------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------

--
To unsubscribe, e-mail: beginners-unsubscribe@perl.org
For additional commands, e-mail: beginners-help@perl.org
http://learn.perl.org/

Re: How to avoid using the slow array subscripting operator?

--bcaec5485cfc57a64704a4754345
Content-Type: text/plain; charset=ISO-8859-1

I guess I should provide some real code.
below is a tiny module I grabbed from my project which is not that privacy
but sufficient for discussion.

#/usr/bin/perl
use warnings;
use 5.010;
use Carp;
use DB_File;

{
my %sum_cache;
my %deviation_cache;
tie %sum_cache => 'DB_File', "sum_cache_tmp", O_RDWR|O_CREAT, 0666 or
croak "can not create cache file!";
tie %deviation_cache => 'DB_File', "deviation_cache_tmp",
O_RDWR|O_CREAT, 0666 or croak "can not create cache file!";


sub get_sum
{
my ($i, $n, $opens_ref) = @_;
my $key = join ',', $i, $n;
unless(exists $sum_cache{$key})
{
my $inner_key = join ',', $i - 1, $n;
my $sum = 0;
if(exists $sum_cache{$inner_key})
{
$sum = $sum_cache{$inner_key} - ${$opens_ref}[$i - $n] +
${$opens_ref}[$i];
}else
{
$sum += $_ foreach @{$opens_ref}[$i - $n + 1 .. $i];
}
$sum_cache{$key} = $sum;
}
return $sum_cache{$key};
}

sub get_deviation
{
my ($i, $n, $mean, $opens_ref) = @_;
my $key = join ',', $i, $n;
unless(exists $deviation_cache{$key})
{
my $sum = 0;
$sum += ($_ - $mean) ** 2 foreach @{$opens_ref}[$i - $n + 1 .. $i];
$deviation_cache{$key} = sqrt($sum / ($n - 1));
}
return $deviation_cache{$key};
}

sub cal_using_simple_scheme
{
my($n1, $n2, $n3, $klines_ref) = @_;
my @opens = map {$_->{open}} @$klines_ref;
croak "Wrong argnument. Should all be positive values" if ($n1 <= 0 || $n2
<= 0 || $n3 <= 0);
return [(0) x @opens] if $n2 <= 1;
my $n = $n1 > $n2 ? $n1 : $n2;
my @res;
my @upper_bound;
my @lower_bound;
for my $i (0 .. $#opens)
{
if($i < $n - 1)
{
push @upper_bound, undef;
push @lower_bound, undef;
next;
}

my $sum1 = get_sum($i, $n1, \@opens);
my $mid = $sum1 / $n1;

my $sum2 = get_sum($i, $n2, \@opens);
my $mean = $sum2 / $n2;
my $std_deviation = get_deviation($i, $n2, $mean, \@opens);

push @upper_bound, $mid + $n3 * $std_deviation;
push @lower_bound, $mid - $n3 * $std_deviation;
}
for my $i (0 .. $#opens)
{
if(!defined($upper_bound[$i]))
{
push @res, 0;
next;
}
given( $opens[$i])
{
when($_ > $upper_bound[$i]) {push @res, 1; break}
when($_ < $lower_bound[$i]) {push @res, -1; break}
default {push @res, $res[-1]; break}
}
}
return \@res;
}

}

1;

and it's called like this:
my $klines_ref;
for my $n1 (1..200)
{
for my $n2 (1..200)
{
for my $n3 (1,3,5,7)
{
my $res = cal_using_simple_scheme($n1, $n2, $n3, $klines_ref);
#using the result to do something, omitted here.
}
}
}

$klines_ref is a reference to an array which contains at least 60000+
elements.
the module is simple, the main time consuming part is calculating sum and
standard deviation part. I cached the results for both sum and
std_deviation. but since the input array is so big that during the first
call of the cal_using_stupid_scheme the program crashes because of out of
memory. So I tie the two hashes to local files, and as I predicted, the
speed is unacceptable. maybe this is the kind of problem perl is good at?
I still use index because I really don't know how to avoid it. would some
body point out the wrong and not perlish part of the program above?
Thank you all.

--bcaec5485cfc57a64704a4754345--