BerkeleyDB Queue Database Array Size
BerkeleyDB Queue Database Array Size
am 14.08.2006 08:50:11 von kallestad
I've been playing around with BerkeleyDB Queue database type, and I've
found something curious.
I'm tying an array variable to the queue database:
$db = tie @samplearray, 'BerkeleyDB::Queue',
-Flags => DB_CREATE,
-Filename => $config->get('db_queue_filename'),
-Env => $env,
-Len => 300
or print "$BerkeleyDB::Error\n" ;
I'm adding records to the database by pushing values into the array:
push @samplearray, $u or die "can't push $u $!\n";
And I'm pulling records from the beginning of the database using shift:
shift @samplearray;
Everything is working brilliantly - although I haven't tried
multiprocess access to the database which is a desired future step.
The curiosity that I have found is this:
$k = $#samplearray;
print "$k \n";
When the array is not "tied" to the BerkeleyDB, this number will shrink
every time I shift a record out of the array - If I pull out 5 of 5
items, it would report 3,2,1,0,-1. However, when it is tied to the
database, it will always report the maximum queue size until there is
nothing left in the queue, at which point it will return -1 like it
normally would if it was just an array. In the above example - it's
4,4,4,4,-1
I tried calling $db->sync after pulling items out, but the issue still
exists. Being a bit of a newbie, I don't know if this is a problem
with the way I am implementing things, my technical understanding, or
if this is an actual bug. Any guidance is appreciated.
Thanks,
Steve
Re: BerkeleyDB Queue Database Array Size
am 14.08.2006 10:38:43 von Paul Marquess
This is a multi-part message in MIME format.
------=_NextPart_000_0068_01C6BF85.6FA02680
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit
> -----Original Message-----
> From: kallestad@gmail.com [mailto:kallestad@gmail.com]
> Posted At: 14 August 2006 07:50
> Posted To: comp.lang.perl.modules
> Conversation: BerkeleyDB Queue Database Array Size
> Subject: BerkeleyDB Queue Database Array Size
>
> I've been playing around with BerkeleyDB Queue database type, and I've
> found something curious.
>
> I'm tying an array variable to the queue database:
> $db = tie @samplearray, 'BerkeleyDB::Queue',
> -Flags => DB_CREATE,
> -Filename => $config->get('db_queue_filename'),
> -Env => $env,
> -Len => 300
> or print "$BerkeleyDB::Error\n" ;
>
> I'm adding records to the database by pushing values into the array:
> push @samplearray, $u or die "can't push $u $!\n";
>
> And I'm pulling records from the beginning of the database using
shift:
> shift @samplearray;
>
> Everything is working brilliantly - although I haven't tried
> multiprocess access to the database which is a desired future step.
> The curiosity that I have found is this:
>
> $k = $#samplearray;
> print "$k \n";
>
> When the array is not "tied" to the BerkeleyDB, this number will
shrink
> every time I shift a record out of the array - If I pull out 5 of 5
> items, it would report 3,2,1,0,-1. However, when it is tied to the
> database, it will always report the maximum queue size until there is
> nothing left in the queue, at which point it will return -1 like it
> normally would if it was just an array. In the above example - it's
> 4,4,4,4,-1
>
> I tried calling $db->sync after pulling items out, but the issue still
> exists. Being a bit of a newbie, I don't know if this is a problem
> with the way I am implementing things, my technical understanding, or
> if this is an actual bug. Any guidance is appreciated.
>
> Thanks,
> Steve
------=_NextPart_000_0068_01C6BF85.6FA02680
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
charset=3Dus-ascii">
6.5.7036.0">
Re: BerkeleyDB Queue Database Array Size
> -----Original Message-----
> From: kallestad@gmail.com [
HREF=3D"mailto:kallestad@gmail.com">mailto:kallestad@gmail.c om]
>
> Posted At: 14 August 2006 07:50
> Posted To: comp.lang.perl.modules
> Conversation: BerkeleyDB Queue Database Array =
Size
> Subject: BerkeleyDB Queue Database Array =
Size
>
> I've been playing around with BerkeleyDB Queue =
database type, and I've
> found something curious.
>
> I'm tying an array variable to the queue =
database:
> $db =3D tie @samplearray, =
'BerkeleyDB::Queue',
> =
-Flags =3D> DB_CREATE,
> =
-Filename =3D> $config->get('db_queue_filename'),
> =
-Env =3D> =
$env,
> =
-Len =3D> =
300
SIZE=3D2>> =
or print "$BerkeleyDB::Error\n" ;
>
> I'm adding records to the database by pushing =
values into the array:
> push @samplearray, $u or die "can't push $u =
$!\n";
>
> And I'm pulling records from the beginning of =
the database using shift:
> shift @samplearray;
>
> Everything is working brilliantly - although I =
haven't tried
> multiprocess access to the database which is a =
desired future step.
> The curiosity that I have found is this:
>
> $k =3D $#samplearray;
> print "$k \n";
>
> When the array is not "tied" to the =
BerkeleyDB, this number will shrink
> every time I shift a record out of the array - =
If I pull out 5 of 5
> items, it would report 3,2,1,0,-1. =
However, when it is tied to the
> database, it will always report the maximum =
queue size until there is
> nothing left in the queue, at which point it =
will return -1 like it
> normally would if it was just an array. In =
the above example - it's
> 4,4,4,4,-1
>
> I tried calling $db->sync after pulling items =
out, but the issue still
> exists. Being a bit of a newbie, I don't =
know if this is a problem
> with the way I am implementing things, my =
technical understanding, or
> if this is an actual bug. Any guidance is =
appreciated.
>
> Thanks,
> Steve
------=_NextPart_000_0068_01C6BF85.6FA02680--
Re: BerkeleyDB Queue Database Array Size
am 14.08.2006 11:00:05 von Paul Marquess
This is a multi-part message in MIME format.
------=_NextPart_000_0071_01C6BF88.6AFC60A0
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit
From: kallestad@gmail.com [mailto:kallestad@gmail.com]
> I've been playing around with BerkeleyDB Queue database type, and I've
> found something curious.
>
> I'm tying an array variable to the queue database:
> $db = tie @samplearray, 'BerkeleyDB::Queue',
> -Flags => DB_CREATE,
> -Filename => $config->get('db_queue_filename'),
> -Env => $env,
> -Len => 300
> or print "$BerkeleyDB::Error\n" ;
>
> I'm adding records to the database by pushing values into the array:
> push @samplearray, $u or die "can't push $u $!\n";
>
> And I'm pulling records from the beginning of the database using
shift:
> shift @samplearray;
>
> Everything is working brilliantly - although I haven't tried
> multiprocess access to the database which is a desired future step.
> The curiosity that I have found is this:
>
> $k = $#samplearray;
> print "$k \n";
>
> When the array is not "tied" to the BerkeleyDB, this number will
shrink
> every time I shift a record out of the array - If I pull out 5 of 5
> items, it would report 3,2,1,0,-1. However, when it is tied to the
> database, it will always report the maximum queue size until there is
> nothing left in the queue, at which point it will return -1 like it
> normally would if it was just an array. In the above example - it's
> 4,4,4,4,-1
>
> I tried calling $db->sync after pulling items out, but the issue still
> exists. Being a bit of a newbie, I don't know if this is a problem
> with the way I am implementing things, my technical understanding, or
> if this is an actual bug. Any guidance is appreciated.
This is a limitation in the current perl interface to Berkeley DB.
The problem with providing an interface to return the number of records
in the queue (or any of the other database types that Berkeley DB
provides) is that the underlying Berkeley DB library has to traverse the
complete database to get an accurate count. If you have a large queue,
this is not desirable.
Have a look at this page. In particular, look at the DB_FAST_STAT entry.
http://www.sleepycat.com/docs/api_c/db_stat.html
If the limitations of DB_FAST_STAT are good enough for your purposes,
you can use it like this
$count = $db->db_stat("qs_nrecs", DB_FAST_STAT);
If you want to take the hit of traversing the database to count the
number of records, this should do the trick
$count = $db->db_stat("qs_nrecs");
Paul
------=_NextPart_000_0071_01C6BF88.6AFC60A0
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
charset=3Dus-ascii">
6.5.7036.0">
Re: BerkeleyDB Queue Database Array Size
From: kallestad@gmail.com [
HREF=3D"mailto:kallestad@gmail.com">mailto:kallestad@gmail.c om]
>
> I've been playing around with BerkeleyDB Queue =
database type, and I've
> found something curious.
>
> I'm tying an array variable to the queue =
database:
> $db =3D tie @samplearray, =
'BerkeleyDB::Queue',
> =
-Flags =3D> DB_CREATE,
> =
-Filename =3D> $config->get('db_queue_filename'),
> =
-Env =3D> =
$env,
> =
-Len =3D> =
300
SIZE=3D2>> =
or print "$BerkeleyDB::Error\n" ;
>
> I'm adding records to the database by pushing =
values into the array:
> push @samplearray, $u or die "can't push $u =
$!\n";
>
> And I'm pulling records from the beginning of =
the database using shift:
> shift @samplearray;
>
> Everything is working brilliantly - although I =
haven't tried
> multiprocess access to the database which is a =
desired future step.
> The curiosity that I have found is this:
>
> $k =3D $#samplearray;
> print "$k \n";
>
> When the array is not "tied" to the =
BerkeleyDB, this number will shrink
> every time I shift a record out of the array - =
If I pull out 5 of 5
> items, it would report 3,2,1,0,-1. =
However, when it is tied to the
> database, it will always report the maximum =
queue size until there is
> nothing left in the queue, at which point it =
will return -1 like it
> normally would if it was just an array. In =
the above example - it's
> 4,4,4,4,-1
>
> I tried calling $db->sync after pulling items =
out, but the issue still
> exists. Being a bit of a newbie, I don't =
know if this is a problem
> with the way I am implementing things, my =
technical understanding, or
> if this is an actual bug. Any guidance is =
appreciated.
This is a limitation in the current perl interface to =
Berkeley DB.
The problem with providing an interface to return the =
number of records in the queue (or any of the other database types that =
Berkeley DB provides) is that the underlying Berkeley DB library has to =
traverse the complete database to get an accurate count. If you have a =
large queue, this is not desirable.
Have a look at this page. In particular, look at the =
DB_FAST_STAT entry.
HREF=3D"http://www.sleepycat.com/docs/api_c/db_stat.html">ht tp://www.slee=
pycat.com/docs/api_c/db_stat.html
If the limitations of DB_FAST_STAT are good enough for =
your purposes, you can use it like this
$count =3D =
$db->db_stat("qs_nrecs", DB_FAST_STAT);
If you want to take the hit of traversing the database =
to count the number of records, this should do the trick
$count =3D =
$db->db_stat("qs_nrecs");
Paul
------=_NextPart_000_0071_01C6BF88.6AFC60A0--
Re: BerkeleyDB Queue Database Array Size
am 14.08.2006 16:38:17 von Paul Marquess
This is a multi-part message in MIME format.
------=_NextPart_000_00D6_01C6BFB7.AA2241D0
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit
From: kallestad@gmail.com [mailto:kallestad@gmail.com]
> I've been playing around with BerkeleyDB Queue database type, and I've
> found something curious.
>
> I'm tying an array variable to the queue database:
> $db = tie @samplearray, 'BerkeleyDB::Queue',
> -Flags => DB_CREATE,
> -Filename => $config->get('db_queue_filename'),
> -Env => $env,
> -Len => 300
> or print "$BerkeleyDB::Error\n" ;
>
> I'm adding records to the database by pushing values into the array:
> push @samplearray, $u or die "can't push $u $!\n";
>
> And I'm pulling records from the beginning of the database using
shift:
> shift @samplearray;
>
> Everything is working brilliantly - although I haven't tried
> multiprocess access to the database which is a desired future step.
> The curiosity that I have found is this:
>
> $k = $#samplearray;
> print "$k \n";
>
> When the array is not "tied" to the BerkeleyDB, this number will
shrink
> every time I shift a record out of the array - If I pull out 5 of 5
> items, it would report 3,2,1,0,-1. However, when it is tied to the
> database, it will always report the maximum queue size until there is
> nothing left in the queue, at which point it will return -1 like it
> normally would if it was just an array. In the above example - it's
> 4,4,4,4,-1
>
> I tried calling $db->sync after pulling items out, but the issue still
> exists. Being a bit of a newbie, I don't know if this is a problem
> with the way I am implementing things, my technical understanding, or
> if this is an actual bug. Any guidance is appreciated.
This is a limitation in the current perl interface to Berkeley DB.
The problem with providing an interface to return the number of records
in the queue (or any of the other database types that Berkeley DB
provides) is that the underlying Berkeley DB library has to traverse the
complete database to get an accurate count. If you have a large queue,
this is not desirable.
Have a look at this page. In particular, look at the DB_FAST_STAT entry.
http://www.sleepycat.com/docs/api_c/db_stat.html
If the limitations of DB_FAST_STAT are good enough for your purposes,
you can use it like this
$count = $db->db_stat("qs_nrecs", DB_FAST_STAT);
If you want to take the hit of traversing the database to count the
number of records, this should do the trick
$count = $db->db_stat("qs_nrecs");
Paul
------=_NextPart_000_00D6_01C6BFB7.AA2241D0
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
charset=3Dus-ascii">
6.5.7036.0">
Re: BerkeleyDB Queue Database Array Size
From: kallestad@gmail.com [
HREF=3D"mailto:kallestad@gmail.com">mailto:kallestad@gmail.c om]
>
> I've been playing around with BerkeleyDB Queue =
database type, and I've
> found something curious.
>
> I'm tying an array variable to the queue =
database:
> $db =3D tie @samplearray, =
'BerkeleyDB::Queue',
> =
-Flags =3D> DB_CREATE,
> =
-Filename =3D> $config->get('db_queue_filename'),
> =
-Env =3D> =
$env,
> =
-Len =3D> =
300
SIZE=3D2>> =
or print "$BerkeleyDB::Error\n" ;
>
> I'm adding records to the database by pushing =
values into the array:
> push @samplearray, $u or die "can't push $u =
$!\n";
>
> And I'm pulling records from the beginning of =
the database using shift:
> shift @samplearray;
>
> Everything is working brilliantly - although I =
haven't tried
> multiprocess access to the database which is a =
desired future step.
> The curiosity that I have found is this:
>
> $k =3D $#samplearray;
> print "$k \n";
>
> When the array is not "tied" to the =
BerkeleyDB, this number will shrink
> every time I shift a record out of the array - =
If I pull out 5 of 5
> items, it would report 3,2,1,0,-1. =
However, when it is tied to the
> database, it will always report the maximum =
queue size until there is
> nothing left in the queue, at which point it =
will return -1 like it
> normally would if it was just an array. In =
the above example - it's
> 4,4,4,4,-1
>
> I tried calling $db->sync after pulling items =
out, but the issue still
> exists. Being a bit of a newbie, I don't =
know if this is a problem
> with the way I am implementing things, my =
technical understanding, or
> if this is an actual bug. Any guidance is =
appreciated.
This is a limitation in the current perl interface to =
Berkeley DB.
The problem with providing an interface to return the =
number of records in the queue (or any of the other database types that =
Berkeley DB provides) is that the underlying Berkeley DB library has to =
traverse the complete database to get an accurate count. If you have a =
large queue, this is not desirable.
Have a look at this page. In particular, look at the =
DB_FAST_STAT entry.
HREF=3D"http://www.sleepycat.com/docs/api_c/db_stat.html">ht tp://www.slee=
pycat.com/docs/api_c/db_stat.html
If the limitations of DB_FAST_STAT are good enough for =
your purposes, you can use it like this
$count =3D =
$db->db_stat("qs_nrecs", DB_FAST_STAT);
If you want to take the hit of traversing the database =
to count the number of records, this should do the trick
$count =3D =
$db->db_stat("qs_nrecs");
Paul
------=_NextPart_000_00D6_01C6BFB7.AA2241D0--
Re: BerkeleyDB Queue Database Array Size
am 14.08.2006 18:06:26 von Paul Marquess
kallestad@gmail.com wrote:
> I've been playing around with BerkeleyDB Queue database type, and I've
> found something curious.
>
> I'm tying an array variable to the queue database:
> $db = tie @samplearray, 'BerkeleyDB::Queue',
> -Flags => DB_CREATE,
> -Filename => $config->get('db_queue_filename'),
> -Env => $env,
> -Len => 300
> or print "$BerkeleyDB::Error\n" ;
>
> I'm adding records to the database by pushing values into the array:
> push @samplearray, $u or die "can't push $u $!\n";
>
> And I'm pulling records from the beginning of the database using shift:
> shift @samplearray;
>
> Everything is working brilliantly - although I haven't tried
> multiprocess access to the database which is a desired future step.
> The curiosity that I have found is this:
>
> $k = $#samplearray;
> print "$k \n";
>
> When the array is not "tied" to the BerkeleyDB, this number will shrink
> every time I shift a record out of the array - If I pull out 5 of 5
> items, it would report 3,2,1,0,-1. However, when it is tied to the
> database, it will always report the maximum queue size until there is
> nothing left in the queue, at which point it will return -1 like it
> normally would if it was just an array. In the above example - it's
> 4,4,4,4,-1
>
> I tried calling $db->sync after pulling items out, but the issue still
> exists. Being a bit of a newbie, I don't know if this is a problem
> with the way I am implementing things, my technical understanding, or
> if this is an actual bug. Any guidance is appreciated
This is a limitation in the current perl interface to Berkeley DB.
The problem with providing an interface to return the number of records in
the queue (or any of the other database types that Berkeley DB provides) is
that the underlying Berkeley DB library has to traverse the complete
database to get an accurate count. If you have a large queue, this is not
desirable.
Have a look at this page. In particular, look at the DB_FAST_STAT entry.
http://www.sleepycat.com/docs/api_c/db_stat.html
If the limitations of DB_FAST_STAT are good enough for your purposes, you
can use it like this
$count = $db->db_stat("qs_nrecs", DB_FAST_STAT);
If you want to take the hit of traversing the database to count the number
of records, this should do the trick
$count = $db->db_stat("qs_nrecs");
cheers
Paul
Re: BerkeleyDB Queue Database Array Size
am 15.08.2006 01:22:29 von kallestad
Paul Marquess wrote:
> kallestad@gmail.com wrote:
>
> > I've been playing around with BerkeleyDB Queue database type, and I've
> > found something curious.
> >
> > I'm tying an array variable to the queue database:
> > $db = tie @samplearray, 'BerkeleyDB::Queue',
> > -Flags => DB_CREATE,
> > -Filename => $config->get('db_queue_filename'),
> > -Env => $env,
> > -Len => 300
> > or print "$BerkeleyDB::Error\n" ;
> >
> > I'm adding records to the database by pushing values into the array:
> > push @samplearray, $u or die "can't push $u $!\n";
> >
> > And I'm pulling records from the beginning of the database using shift:
> > shift @samplearray;
> >
> > Everything is working brilliantly - although I haven't tried
> > multiprocess access to the database which is a desired future step.
> > The curiosity that I have found is this:
> >
> > $k = $#samplearray;
> > print "$k \n";
> >
> > When the array is not "tied" to the BerkeleyDB, this number will shrink
> > every time I shift a record out of the array - If I pull out 5 of 5
> > items, it would report 3,2,1,0,-1. However, when it is tied to the
> > database, it will always report the maximum queue size until there is
> > nothing left in the queue, at which point it will return -1 like it
> > normally would if it was just an array. In the above example - it's
> > 4,4,4,4,-1
> >
> > I tried calling $db->sync after pulling items out, but the issue still
> > exists. Being a bit of a newbie, I don't know if this is a problem
> > with the way I am implementing things, my technical understanding, or
> > if this is an actual bug. Any guidance is appreciated
>
> This is a limitation in the current perl interface to Berkeley DB.
>
> The problem with providing an interface to return the number of records in
> the queue (or any of the other database types that Berkeley DB provides) is
> that the underlying Berkeley DB library has to traverse the complete
> database to get an accurate count. If you have a large queue, this is not
> desirable.
>
> Have a look at this page. In particular, look at the DB_FAST_STAT entry.
>
> http://www.sleepycat.com/docs/api_c/db_stat.html
>
> If the limitations of DB_FAST_STAT are good enough for your purposes, you
> can use it like this
>
>
> $count = $db->db_stat("qs_nrecs", DB_FAST_STAT);
>
> If you want to take the hit of traversing the database to count the number
> of records, this should do the trick
>
> $count = $db->db_stat("qs_nrecs");
>
> cheers
> Paul
Thanks for the help! It could be that I'm working with a slightly
older versions, but the above syntax didn't quite work for me. With
your guidance, I came up with the following:
$k = $db->db_stat(DB_FAST_STAT);
$num = $k->{'qs_cur_recno'}-$k->{'qs_first_recno'} ;
Perl complained that db_stat wanted only a numeric argument. When I
took a look at the hash that was returned, I didn't have a "qs_nrecs"
key, but the information I was looking for was easily discerned by
subtracting the first record number (qs_first_recno) from the current
record number(qs_cur_recno).
I appreciate your taking the time to help me out.