perl dbi memory error?

--0-1380304430-1176714988=:81153
Content-Type: text/plain; charset=ascii

Hai,

I am using perl DBI module for fetching data from database.



My database table contains almost >70 million entries.

I am fetching the data using following steps

1. $dbh->DBI->connect(...);

2.
$sth = $dbh->prepare("Select * from $TABLE") //which contains more
than >70 million entries

3. $dbh->execute

4. $sth->fetchrow_array , to fetch data




My question is because of the large data (>70 million entries) , any
memory error like memory out of error comes (because i observed at
some instance memory utilization > 70 %).



Basically i want to know , the fetch command gets from db one row at a time or it gets from the memory

Any
better approach to make it use less memory . I tried by reading entries
in chunks of(10,000) , memory is low but speed is slow...

Thanks for your help

Thanks,
N Ravi


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
--0-1380304430-1176714988=:81153--

Re: perl dbi memory error?

Sort answer "Yes with a But.."
Long answer "No with a Maybe.."

I guess it would depend on the DBD client you are using. Some are more
capable than others. For example with DBD::Oracle you can create a cursor
on the server sideg and then get that in clumps.

Give us the exact DBD clients you are using and I am sure one of out here
can give you a number of examples.

cheers
----- Original Message -----
From: "ravi kumar"
To:
Sent: Monday, April 16, 2007 5:16 AM
Subject: perl dbi memory error?


> Hai,
>
> I am using perl DBI module for fetching data from database.
>
>
>
> My database table contains almost >70 million entries.
>
> I am fetching the data using following steps
>
> 1. $dbh->DBI->connect(...);
>
> 2.
> $sth = $dbh->prepare("Select * from $TABLE") //which contains more
> than >70 million entries
>
> 3. $dbh->execute
>
> 4. $sth->fetchrow_array , to fetch data
>
>
>
>
> My question is because of the large data (>70 million entries) , any
> memory error like memory out of error comes (because i observed at
> some instance memory utilization > 70 %).
>
>
>
> Basically i want to know , the fetch command gets from db one row at a
> time or it gets from the memory
>
> Any
> better approach to make it use less memory . I tried by reading entries
> in chunks of(10,000) , memory is low but speed is slow...
>
> Thanks for your help
>
> Thanks,
> N Ravi
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com

Re: perl dbi memory error?

By the way: In the 3rd step You are using in Your code $sth->execute (and not
$dbh->execute, as You typed in Your message)? Don't You?

Best regards

Robert

ravi kumar schrieb:
> Hai,
>
> I am using perl DBI module for fetching data from database.
>
>
>
> My database table contains almost >70 million entries.
>
> I am fetching the data using following steps
>
> 1. $dbh->DBI->connect(...);
>
> 2.
> $sth = $dbh->prepare("Select * from $TABLE") //which contains more
> than >70 million entries
>
> 3. $dbh->execute
>
> 4. $sth->fetchrow_array , to fetch data
>
>
>
>
> My question is because of the large data (>70 million entries) , any
> memory error like memory out of error comes (because i observed at
> some instance memory utilization > 70 %).
>
>
>
> Basically i want to know , the fetch command gets from db one row at a time or it gets from the memory
>
> Any
> better approach to make it use less memory . I tried by reading entries
> in chunks of(10,000) , memory is low but speed is slow...
>
> Thanks for your help
>
> Thanks,
> N Ravi
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com

Re: perl dbi memory error?

--0-210574892-1176961749=:50840
Content-Type: text/plain; charset=ascii

I am using mysql database.

Re: perl dbi memory error?





Thread Previous








From:
John Scoles

Date:
April 16, 2007 04:35

Subject:
Re: perl dbi memory error?




Sort answer "Yes with a But.."
Long answer "No with a Maybe.."
I guess it would depend on the DBD client you are using. Some are more
capable than others. For example with DBD::Oracle you can create a cursor
on the server sideg and then get that in clumps.

Give us the exact DBD clients you are using and I am sure one of out here
can give you a number of examples.

cheers
----- Original Message -----
From: "ravi kumar"
To:
Sent: Monday, April 16, 2007 5:16 AM
Subject: perl dbi memory error?


> Hai,
>
> I am using perl DBI module for fetching data from database.
>
>
>
> My database table contains almost >70 million entries.
>
> I am fetching the data using following steps
>
> 1. $dbh->DBI->connect(...);
>
> 2.
> $sth = $dbh->prepare("Select * from $TABLE") //which contains more
> than >70 million entries
>
> 3. $dbh->execute
>
> 4. $sth->fetchrow_array , to fetch data
>
>
>
>
> My question is because of the large data (>70 million entries) , any
> memory error like memory out of error comes (because i observed at
> some instance memory utilization > 70 %).
>
>
>
> Basically i want to know , the fetch command gets from db one row at a
> time or it gets from the memory
>
> Any
> better approach to make it use less memory . I tried by reading entries
> in chunks of(10,000) , memory is low but speed is slow...
>
> Thanks for your help
>
> Thanks,
> N Ravi
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com





Thread Previous








perl dbi memory error?
by ravi kumar
Re: perl dbi memory error?
by Robert Roggenbuck
Re: perl dbi memory error?
by John Scoles



__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
--0-210574892-1176961749=:50840--