Tie::File surprise (to me)

I probably should have known this, but didn't. In Tie::File, where you
tie an array to a long file, and you want to read backwards from the end
of it ( to look for the most recent lines, for example, in a log file),
it is much faster to address the lines with $index as a negative number,
meaning to search from the end of the array. It takes a while for the
module to find the index = -1, but the subsequent searches on -2, -3,
etc. are much faster than using @#array-1, @#array-2, etc.

This may be optimized in later versions - I am using 0.97 version of
Tie::File.

The times I got were on a log file of 30000 lines or so: (my target
lines were ones that started with /^From: / )

Finding last line index, the first way, took: 1.21123 seconds
Finding the "from" lines, the first way, took: 1.86017 seconds
Finding last line index, the second way, took: 1.01147 seconds
Finding the "from" lines, the second way, took: 0.0000119 seconds

where "first way" is using @#array value and decrementing it,
the "second way" is using -1, -2, ...

Finding the last line is about the same in both cases (varies from run
to run) (I used Time::HiRes to get the times - not as accurate as
Devel::DProf, but easier to use in my script).

But notice the tremendous difference in the time of finding the "from"
lines.

Comments?

Boyd

Re: Tie::File surprise (to me)

In article ,
boyd wrote:

> I probably should have known this, but didn't. In Tie::File, where you
> tie an array to a long file, and you want to read backwards from the end
> of it ( to look for the most recent lines, for example, in a log file),
> it is much faster to address the lines with $index as a negative number,
> meaning to search from the end of the array. It takes a while for the
> module to find the index = -1, but the subsequent searches on -2, -3,
> etc. are much faster than using @#array-1, @#array-2, etc.
>
> This may be optimized in later versions - I am using 0.97 version of
> Tie::File.
>
> The times I got were on a log file of 30000 lines or so: (my target
> lines were ones that started with /^From: / )
>
> Finding last line index, the first way, took: 1.21123 seconds
> Finding the "from" lines, the first way, took: 1.86017 seconds
> Finding last line index, the second way, took: 1.01147 seconds
> Finding the "from" lines, the second way, took: 0.0000119 seconds
>
> where "first way" is using @#array value and decrementing it,
> the "second way" is using -1, -2, ...
>
> Finding the last line is about the same in both cases (varies from run
> to run) (I used Time::HiRes to get the times - not as accurate as
> Devel::DProf, but easier to use in my script).
>
> But notice the tremendous difference in the time of finding the "from"
> lines.
>
> Comments?
>
> Boyd

Ignore this posting - I made a dumb error in my second loop - it was not
really running at all. :-!

Boyd

Re: Tie::File surprise (to me)

>>>>> "b" == boyd writes:

b> I probably should have known this, but didn't. In Tie::File, where you
b> tie an array to a long file, and you want to read backwards from the end
b> of it ( to look for the most recent lines, for example, in a log file),
b> it is much faster to address the lines with $index as a negative number,
b> meaning to search from the end of the array. It takes a while for the
b> module to find the index = -1, but the subsequent searches on -2, -3,
b> etc. are much faster than using @#array-1, @#array-2, etc.

you should use File::ReadBackwards for that. much faster than tying a
log file.

uri

--
Uri Guttman ------ uri@stemsystems.com -------- http://www.stemsystems.com
--Perl Consulting, Stem Development, Systems Architecture, Design and Coding-
Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org

Re: Tie::File surprise (to me)

In article ,
Uri Guttman wrote:

> >>>>> "b" == boyd writes:
>
> b> I probably should have known this, but didn't. In Tie::File, where you
> b> tie an array to a long file, and you want to read backwards from the end
> b> of it ( to look for the most recent lines, for example, in a log file),
> b> it is much faster to address the lines with $index as a negative number,
> b> meaning to search from the end of the array. It takes a while for the
> b> module to find the index = -1, but the subsequent searches on -2, -3,
> b> etc. are much faster than using @#array-1, @#array-2, etc.
>
> you should use File::ReadBackwards for that. much faster than tying a
> log file.
>
> uri

Thanks, uri. I didn't know about File::ReadBackwards.

Boyd