Issue with the huge import script

--000feaf54f78640694047138049a
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi,

I have to import data from one database to another, I have to import around
100000(1Lac) records.
First I need to check if the record is already imported or not and import
only those records which are not imported.

Here is my logic

$already_imported = get_already_imported_records();
format of the $already_imported is $already_imported[someid] = 'imported';

Now i take all records from another db and iterating through it.

if (!key_exists($already_imported[$new_id])){
import_function($new_id)
}else{
echo 'allready imported'.$already_imported[$new_id];
}

Now my script is importing same records for more than one time. I am not
able to get through this issue

Is it because of the size of the records or something else...?

Please suggest me some solution which is faster, safe and easy to code :D

Thanks in advance

--
Devendra Jadhav

--000feaf54f78640694047138049a--

Re: Issue with the huge import script

Hi,

this sounds huge, and cries for a sql version of the import.
Are both databases the same? MySQL?

I give u a draft for MySQL
u export the data u have, then u got a textfile with 100000+ sql statments

in the php script u open the file and iterate over it by line (carefull it
could be also ";" in case its a Unix created file on a windows platform)

line == one SQL insert in table bla bla...

in the loop then just mysq_query with this line

if the the someid is an unique index the insert will fail, so only those
records are inserted beeing not already in the database.

but I think as of the amount off records it doesn't sound like a every 10
minutes job, if it is a rara job, just do it with phpMyAdmin

sorry not pulling out the code, but was a long day behind the keyboard, need
some sleep

ralph_deffke@yahoo.de




"Devendra Jadhav" wrote in message
news:be4b00cf0908151815r1c7430d2j8a6cb0da1f10ac00@mail.gmail .com...
> Hi,
>
> I have to import data from one database to another, I have to import
around
> 100000(1Lac) records.
> First I need to check if the record is already imported or not and import
> only those records which are not imported.
>
> Here is my logic
>
> $already_imported = get_already_imported_records();
> format of the $already_imported is $already_imported[someid] = 'imported';
>
> Now i take all records from another db and iterating through it.
>
> if (!key_exists($already_imported[$new_id])){
> import_function($new_id)
> }else{
> echo 'allready imported'.$already_imported[$new_id];
> }
>
> Now my script is importing same records for more than one time. I am not
> able to get through this issue
>
> Is it because of the size of the records or something else...?
>
> Please suggest me some solution which is faster, safe and easy to code :D
>
> Thanks in advance
>
> --
> Devendra Jadhav
>



--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Issue with the huge import script

On Sun, 2009-08-16 at 04:06 +0200, Ralph Deffke wrote:
> Hi,
>
> this sounds huge, and cries for a sql version of the import.
> Are both databases the same? MySQL?
>
> I give u a draft for MySQL
> u export the data u have, then u got a textfile with 100000+ sql statments
>
> in the php script u open the file and iterate over it by line (carefull it
> could be also ";" in case its a Unix created file on a windows platform)
>
> line == one SQL insert in table bla bla...
>
> in the loop then just mysq_query with this line
>
> if the the someid is an unique index the insert will fail, so only those
> records are inserted beeing not already in the database.
>
> but I think as of the amount off records it doesn't sound like a every 10
> minutes job, if it is a rara job, just do it with phpMyAdmin
>
> sorry not pulling out the code, but was a long day behind the keyboard, need
> some sleep
>
> ralph_deffke@yahoo.de
>
>
>
>
> "Devendra Jadhav" wrote in message
> news:be4b00cf0908151815r1c7430d2j8a6cb0da1f10ac00@mail.gmail .com...
> > Hi,
> >
> > I have to import data from one database to another, I have to import
> around
> > 100000(1Lac) records.
> > First I need to check if the record is already imported or not and import
> > only those records which are not imported.
> >
> > Here is my logic
> >
> > $already_imported = get_already_imported_records();
> > format of the $already_imported is $already_imported[someid] = 'imported';
> >
> > Now i take all records from another db and iterating through it.
> >
> > if (!key_exists($already_imported[$new_id])){
> > import_function($new_id)
> > }else{
> > echo 'allready imported'.$already_imported[$new_id];
> > }
> >
> > Now my script is importing same records for more than one time. I am not
> > able to get through this issue
> >
> > Is it because of the size of the records or something else...?
> >
> > Please suggest me some solution which is faster, safe and easy to code :D
> >
> > Thanks in advance
> >
> > --
> > Devendra Jadhav
> >
>
>
>
You cry for a MySQL version and then revert back to PHP?! Why not just
keep the whole thing in MySQL? You can use SQL statements to check
whether a record exists before attempting to shove it in the database
using a WHERE clause in the INSERT statement or by making one field
unique and hiding notices about inserts that are attempting to overwrite
that.

Thanks,
Ash
http://www.ashleysheridan.co.uk


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Issue with the huge import script

because I assume always that a requester got some intelligence, so in that
case there must be a reasonsble reason why he wants to do it in PHP

im not like u assuming everybody is a thumb

ralph

"Ashley Sheridan" wrote in message
news:1250413427.2344.51.camel@localhost...
> On Sun, 2009-08-16 at 04:06 +0200, Ralph Deffke wrote:
> > Hi,
> >
> > this sounds huge, and cries for a sql version of the import.
> > Are both databases the same? MySQL?
> >
> > I give u a draft for MySQL
> > u export the data u have, then u got a textfile with 100000+ sql
statments
> >
> > in the php script u open the file and iterate over it by line (carefull
it
> > could be also ";" in case its a Unix created file on a windows platform)
> >
> > line == one SQL insert in table bla bla...
> >
> > in the loop then just mysq_query with this line
> >
> > if the the someid is an unique index the insert will fail, so only those
> > records are inserted beeing not already in the database.
> >
> > but I think as of the amount off records it doesn't sound like a every
10
> > minutes job, if it is a rara job, just do it with phpMyAdmin
> >
> > sorry not pulling out the code, but was a long day behind the keyboard,
need
> > some sleep
> >
> > ralph_deffke@yahoo.de
> >
> >
> >
> >
> > "Devendra Jadhav" wrote in message
> > news:be4b00cf0908151815r1c7430d2j8a6cb0da1f10ac00@mail.gmail .com...
> > > Hi,
> > >
> > > I have to import data from one database to another, I have to import
> > around
> > > 100000(1Lac) records.
> > > First I need to check if the record is already imported or not and
import
> > > only those records which are not imported.
> > >
> > > Here is my logic
> > >
> > > $already_imported = get_already_imported_records();
> > > format of the $already_imported is $already_imported[someid] =
'imported';
> > >
> > > Now i take all records from another db and iterating through it.
> > >
> > > if (!key_exists($already_imported[$new_id])){
> > > import_function($new_id)
> > > }else{
> > > echo 'allready imported'.$already_imported[$new_id];
> > > }
> > >
> > > Now my script is importing same records for more than one time. I am
not
> > > able to get through this issue
> > >
> > > Is it because of the size of the records or something else...?
> > >
> > > Please suggest me some solution which is faster, safe and easy to code
:D
> > >
> > > Thanks in advance
> > >
> > > --
> > > Devendra Jadhav
> > >
> >
> >
> >
> You cry for a MySQL version and then revert back to PHP?! Why not just
> keep the whole thing in MySQL? You can use SQL statements to check
> whether a record exists before attempting to shove it in the database
> using a WHERE clause in the INSERT statement or by making one field
> unique and hiding notices about inserts that are attempting to overwrite
> that.
>
> Thanks,
> Ash
> http://www.ashleysheridan.co.uk
>



--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Issue with the huge import script

On Sun, 2009-08-16 at 11:05 +0200, Ralph Deffke wrote:
> because I assume always that a requester got some intelligence, so in that
> case there must be a reasonsble reason why he wants to do it in PHP
>
> im not like u assuming everybody is a thumb
>
> ralph
>
> "Ashley Sheridan" wrote in message
> news:1250413427.2344.51.camel@localhost...
> > On Sun, 2009-08-16 at 04:06 +0200, Ralph Deffke wrote:
> > > Hi,
> > >
> > > this sounds huge, and cries for a sql version of the import.
> > > Are both databases the same? MySQL?
> > >
> > > I give u a draft for MySQL
> > > u export the data u have, then u got a textfile with 100000+ sql
> statments
> > >
> > > in the php script u open the file and iterate over it by line (carefull
> it
> > > could be also ";" in case its a Unix created file on a windows platform)
> > >
> > > line == one SQL insert in table bla bla...
> > >
> > > in the loop then just mysq_query with this line
> > >
> > > if the the someid is an unique index the insert will fail, so only those
> > > records are inserted beeing not already in the database.
> > >
> > > but I think as of the amount off records it doesn't sound like a every
> 10
> > > minutes job, if it is a rara job, just do it with phpMyAdmin
> > >
> > > sorry not pulling out the code, but was a long day behind the keyboard,
> need
> > > some sleep
> > >
> > > ralph_deffke@yahoo.de
> > >
> > >
> > >
> > >
> > > "Devendra Jadhav" wrote in message
> > > news:be4b00cf0908151815r1c7430d2j8a6cb0da1f10ac00@mail.gmail .com...
> > > > Hi,
> > > >
> > > > I have to import data from one database to another, I have to import
> > > around
> > > > 100000(1Lac) records.
> > > > First I need to check if the record is already imported or not and
> import
> > > > only those records which are not imported.
> > > >
> > > > Here is my logic
> > > >
> > > > $already_imported = get_already_imported_records();
> > > > format of the $already_imported is $already_imported[someid] =
> 'imported';
> > > >
> > > > Now i take all records from another db and iterating through it.
> > > >
> > > > if (!key_exists($already_imported[$new_id])){
> > > > import_function($new_id)
> > > > }else{
> > > > echo 'allready imported'.$already_imported[$new_id];
> > > > }
> > > >
> > > > Now my script is importing same records for more than one time. I am
> not
> > > > able to get through this issue
> > > >
> > > > Is it because of the size of the records or something else...?
> > > >
> > > > Please suggest me some solution which is faster, safe and easy to code
> :D
> > > >
> > > > Thanks in advance
> > > >
> > > > --
> > > > Devendra Jadhav
> > > >
> > >
> > >
> > >
> > You cry for a MySQL version and then revert back to PHP?! Why not just
> > keep the whole thing in MySQL? You can use SQL statements to check
> > whether a record exists before attempting to shove it in the database
> > using a WHERE clause in the INSERT statement or by making one field
> > unique and hiding notices about inserts that are attempting to overwrite
> > that.
> >
> > Thanks,
> > Ash
> > http://www.ashleysheridan.co.uk
> >
>
>
>
There are rare occasions on this list where the best answer is not PHP,
and I believe this is one of them.

Thanks,
Ash
http://www.ashleysheridan.co.uk


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Issue with the huge import script

so then tell me tell me what my first sentence means

"this sounds huge, and cries for a sql version of the import."

it looks like u have no experience in working as consultant hotlines Ash
first the folk is asking for a sulotion of HIS php implying a general
solution in PHP with a hint to the prof solution

I think thats what I did


"Ashley Sheridan" wrote in message
news:1250414284.2344.55.camel@localhost...
> On Sun, 2009-08-16 at 11:05 +0200, Ralph Deffke wrote:
> > because I assume always that a requester got some intelligence, so in
that
> > case there must be a reasonsble reason why he wants to do it in PHP
> >
> > im not like u assuming everybody is a thumb
> >
> > ralph
> >
> > "Ashley Sheridan" wrote in message
> > news:1250413427.2344.51.camel@localhost...
> > > On Sun, 2009-08-16 at 04:06 +0200, Ralph Deffke wrote:
> > > > Hi,
> > > >
> > > > this sounds huge, and cries for a sql version of the import.
> > > > Are both databases the same? MySQL?
> > > >
> > > > I give u a draft for MySQL
> > > > u export the data u have, then u got a textfile with 100000+ sql
> > statments
> > > >
> > > > in the php script u open the file and iterate over it by line
(carefull
> > it
> > > > could be also ";" in case its a Unix created file on a windows
platform)
> > > >
> > > > line == one SQL insert in table bla bla...
> > > >
> > > > in the loop then just mysq_query with this line
> > > >
> > > > if the the someid is an unique index the insert will fail, so only
those
> > > > records are inserted beeing not already in the database.
> > > >
> > > > but I think as of the amount off records it doesn't sound like a
every
> > 10
> > > > minutes job, if it is a rara job, just do it with phpMyAdmin
> > > >
> > > > sorry not pulling out the code, but was a long day behind the
keyboard,
> > need
> > > > some sleep
> > > >
> > > > ralph_deffke@yahoo.de
> > > >
> > > >
> > > >
> > > >
> > > > "Devendra Jadhav" wrote in message
> > > > news:be4b00cf0908151815r1c7430d2j8a6cb0da1f10ac00@mail.gmail .com...
> > > > > Hi,
> > > > >
> > > > > I have to import data from one database to another, I have to
import
> > > > around
> > > > > 100000(1Lac) records.
> > > > > First I need to check if the record is already imported or not and
> > import
> > > > > only those records which are not imported.
> > > > >
> > > > > Here is my logic
> > > > >
> > > > > $already_imported = get_already_imported_records();
> > > > > format of the $already_imported is $already_imported[someid] =
> > 'imported';
> > > > >
> > > > > Now i take all records from another db and iterating through it.
> > > > >
> > > > > if (!key_exists($already_imported[$new_id])){
> > > > > import_function($new_id)
> > > > > }else{
> > > > > echo 'allready imported'.$already_imported[$new_id];
> > > > > }
> > > > >
> > > > > Now my script is importing same records for more than one time. I
am
> > not
> > > > > able to get through this issue
> > > > >
> > > > > Is it because of the size of the records or something else...?
> > > > >
> > > > > Please suggest me some solution which is faster, safe and easy to
code
> > :D
> > > > >
> > > > > Thanks in advance
> > > > >
> > > > > --
> > > > > Devendra Jadhav
> > > > >
> > > >
> > > >
> > > >
> > > You cry for a MySQL version and then revert back to PHP?! Why not just
> > > keep the whole thing in MySQL? You can use SQL statements to check
> > > whether a record exists before attempting to shove it in the database
> > > using a WHERE clause in the INSERT statement or by making one field
> > > unique and hiding notices about inserts that are attempting to
overwrite
> > > that.
> > >
> > > Thanks,
> > > Ash
> > > http://www.ashleysheridan.co.uk
> > >
> >
> >
> >
> There are rare occasions on this list where the best answer is not PHP,
> and I believe this is one of them.
>
> Thanks,
> Ash
> http://www.ashleysheridan.co.uk
>



--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Issue with the huge import script

On Sun, 2009-08-16 at 11:25 +0200, Ralph Deffke wrote:
> so then tell me tell me what my first sentence means
>
> "this sounds huge, and cries for a sql version of the import."
>
> it looks like u have no experience in working as consultant hotlines Ash
> first the folk is asking for a sulotion of HIS php implying a general
> solution in PHP with a hint to the prof solution
>
> I think thats what I did
>
>
> "Ashley Sheridan" wrote in message
> news:1250414284.2344.55.camel@localhost...
> > On Sun, 2009-08-16 at 11:05 +0200, Ralph Deffke wrote:
> > > because I assume always that a requester got some intelligence, so in
> that
> > > case there must be a reasonsble reason why he wants to do it in PHP
> > >
> > > im not like u assuming everybody is a thumb
> > >
> > > ralph
> > >
> > > "Ashley Sheridan" wrote in message
> > > news:1250413427.2344.51.camel@localhost...
> > > > On Sun, 2009-08-16 at 04:06 +0200, Ralph Deffke wrote:
> > > > > Hi,
> > > > >
> > > > > this sounds huge, and cries for a sql version of the import.
> > > > > Are both databases the same? MySQL?
> > > > >
> > > > > I give u a draft for MySQL
> > > > > u export the data u have, then u got a textfile with 100000+ sql
> > > statments
> > > > >
> > > > > in the php script u open the file and iterate over it by line
> (carefull
> > > it
> > > > > could be also ";" in case its a Unix created file on a windows
> platform)
> > > > >
> > > > > line == one SQL insert in table bla bla...
> > > > >
> > > > > in the loop then just mysq_query with this line
> > > > >
> > > > > if the the someid is an unique index the insert will fail, so only
> those
> > > > > records are inserted beeing not already in the database.
> > > > >
> > > > > but I think as of the amount off records it doesn't sound like a
> every
> > > 10
> > > > > minutes job, if it is a rara job, just do it with phpMyAdmin
> > > > >
> > > > > sorry not pulling out the code, but was a long day behind the
> keyboard,
> > > need
> > > > > some sleep
> > > > >
> > > > > ralph_deffke@yahoo.de
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > "Devendra Jadhav" wrote in message
> > > > > news:be4b00cf0908151815r1c7430d2j8a6cb0da1f10ac00@mail.gmail .com...
> > > > > > Hi,
> > > > > >
> > > > > > I have to import data from one database to another, I have to
> import
> > > > > around
> > > > > > 100000(1Lac) records.
> > > > > > First I need to check if the record is already imported or not and
> > > import
> > > > > > only those records which are not imported.
> > > > > >
> > > > > > Here is my logic
> > > > > >
> > > > > > $already_imported = get_already_imported_records();
> > > > > > format of the $already_imported is $already_imported[someid] =
> > > 'imported';
> > > > > >
> > > > > > Now i take all records from another db and iterating through it.
> > > > > >
> > > > > > if (!key_exists($already_imported[$new_id])){
> > > > > > import_function($new_id)
> > > > > > }else{
> > > > > > echo 'allready imported'.$already_imported[$new_id];
> > > > > > }
> > > > > >
> > > > > > Now my script is importing same records for more than one time. I
> am
> > > not
> > > > > > able to get through this issue
> > > > > >
> > > > > > Is it because of the size of the records or something else...?
> > > > > >
> > > > > > Please suggest me some solution which is faster, safe and easy to
> code
> > > :D
> > > > > >
> > > > > > Thanks in advance
> > > > > >
> > > > > > --
> > > > > > Devendra Jadhav
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > You cry for a MySQL version and then revert back to PHP?! Why not just
> > > > keep the whole thing in MySQL? You can use SQL statements to check
> > > > whether a record exists before attempting to shove it in the database
> > > > using a WHERE clause in the INSERT statement or by making one field
> > > > unique and hiding notices about inserts that are attempting to
> overwrite
> > > > that.
> > > >
> > > > Thanks,
> > > > Ash
> > > > http://www.ashleysheridan.co.uk
> > > >
> > >
> > >
> > >
> > There are rare occasions on this list where the best answer is not PHP,
> > and I believe this is one of them.
> >
> > Thanks,
> > Ash
> > http://www.ashleysheridan.co.uk
> >
>
>
>
You did say this cries for a pure SQL solution, which you then went on
to say involved PHP. Call me a pedant, but PHP is not SQL. And please,
try to leave insults out of the list in future, it makes you look
unprofessional.

Thanks,
Ash
http://www.ashleysheridan.co.uk


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Issue with the huge import script

On Sun, 16 Aug 2009 04:15:12 +0300, Devendra Jadhav
wrote:

> Is it because of the size of the records or something else...?
>
> Please suggest me some solution which is faster, safe and easy to code :D
>

What about using insert ignore ? :)

1. add UNIQUE constraint on a key you want to add, so you won't be able to
add row with same ID twice.
2. then use INSERT IGNORE into ....

=> easy to code, easy to use.

Of course, this aproach won't work if database will grow larger and
larger. (if you adding new rows all the time)

Then it is better to use this aproach:

1. add auto_increment column to original table.
2. before getting new rows from old table, check what is maximum ID from
new table.
3. get only rows where old_table.id > new_table.max_id remember to ORDER
BY ID ASC.



--

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Issue with the huge import script

better consider if u r insulting I've seen a view posts of u falling into
that category i w'ld recomment respect posters first approaches even if they
r stupid and incorporate the respect in the way u answer

the comment "you r crying ..." is an insult.

"Ashley Sheridan" wrote in message
news:1250417781.2344.58.camel@localhost...
> On Sun, 2009-08-16 at 11:25 +0200, Ralph Deffke wrote:
> > so then tell me tell me what my first sentence means
> >
> > "this sounds huge, and cries for a sql version of the import."
> >
> > it looks like u have no experience in working as consultant hotlines
Ash
> > first the folk is asking for a sulotion of HIS php implying a general
> > solution in PHP with a hint to the prof solution
> >
> > I think thats what I did
> >
> >
> > "Ashley Sheridan" wrote in message
> > news:1250414284.2344.55.camel@localhost...
> > > On Sun, 2009-08-16 at 11:05 +0200, Ralph Deffke wrote:
> > > > because I assume always that a requester got some intelligence, so
in
> > that
> > > > case there must be a reasonsble reason why he wants to do it in PHP
> > > >
> > > > im not like u assuming everybody is a thumb
> > > >
> > > > ralph
> > > >
> > > > "Ashley Sheridan" wrote in message
> > > > news:1250413427.2344.51.camel@localhost...
> > > > > On Sun, 2009-08-16 at 04:06 +0200, Ralph Deffke wrote:
> > > > > > Hi,
> > > > > >
> > > > > > this sounds huge, and cries for a sql version of the import.
> > > > > > Are both databases the same? MySQL?
> > > > > >
> > > > > > I give u a draft for MySQL
> > > > > > u export the data u have, then u got a textfile with 100000+ sql
> > > > statments
> > > > > >
> > > > > > in the php script u open the file and iterate over it by line
> > (carefull
> > > > it
> > > > > > could be also ";" in case its a Unix created file on a windows
> > platform)
> > > > > >
> > > > > > line == one SQL insert in table bla bla...
> > > > > >
> > > > > > in the loop then just mysq_query with this line
> > > > > >
> > > > > > if the the someid is an unique index the insert will fail, so
only
> > those
> > > > > > records are inserted beeing not already in the database.
> > > > > >
> > > > > > but I think as of the amount off records it doesn't sound like a
> > every
> > > > 10
> > > > > > minutes job, if it is a rara job, just do it with phpMyAdmin
> > > > > >
> > > > > > sorry not pulling out the code, but was a long day behind the
> > keyboard,
> > > > need
> > > > > > some sleep
> > > > > >
> > > > > > ralph_deffke@yahoo.de
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > "Devendra Jadhav" wrote in message
> > > > > >
news:be4b00cf0908151815r1c7430d2j8a6cb0da1f10ac00@mail.gmail .com...
> > > > > > > Hi,
> > > > > > >
> > > > > > > I have to import data from one database to another, I have to
> > import
> > > > > > around
> > > > > > > 100000(1Lac) records.
> > > > > > > First I need to check if the record is already imported or not
and
> > > > import
> > > > > > > only those records which are not imported.
> > > > > > >
> > > > > > > Here is my logic
> > > > > > >
> > > > > > > $already_imported = get_already_imported_records();
> > > > > > > format of the $already_imported is $already_imported[someid] =
> > > > 'imported';
> > > > > > >
> > > > > > > Now i take all records from another db and iterating through
it.
> > > > > > >
> > > > > > > if (!key_exists($already_imported[$new_id])){
> > > > > > > import_function($new_id)
> > > > > > > }else{
> > > > > > > echo 'allready imported'.$already_imported[$new_id];
> > > > > > > }
> > > > > > >
> > > > > > > Now my script is importing same records for more than one
time. I
> > am
> > > > not
> > > > > > > able to get through this issue
> > > > > > >
> > > > > > > Is it because of the size of the records or something else...?
> > > > > > >
> > > > > > > Please suggest me some solution which is faster, safe and easy
to
> > code
> > > > :D
> > > > > > >
> > > > > > > Thanks in advance
> > > > > > >
> > > > > > > --
> > > > > > > Devendra Jadhav
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > You cry for a MySQL version and then revert back to PHP?! Why not
just
> > > > > keep the whole thing in MySQL? You can use SQL statements to check
> > > > > whether a record exists before attempting to shove it in the
database
> > > > > using a WHERE clause in the INSERT statement or by making one
field
> > > > > unique and hiding notices about inserts that are attempting to
> > overwrite
> > > > > that.
> > > > >
> > > > > Thanks,
> > > > > Ash
> > > > > http://www.ashleysheridan.co.uk
> > > > >
> > > >
> > > >
> > > >
> > > There are rare occasions on this list where the best answer is not
PHP,
> > > and I believe this is one of them.
> > >
> > > Thanks,
> > > Ash
> > > http://www.ashleysheridan.co.uk
> > >
> >
> >
> >
> You did say this cries for a pure SQL solution, which you then went on
> to say involved PHP. Call me a pedant, but PHP is not SQL. And please,
> try to leave insults out of the list in future, it makes you look
> unprofessional.
>
> Thanks,
> Ash
> http://www.ashleysheridan.co.uk
>



--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Issue with the huge import script

On Aug 15, 2009, at 9:15 PM, Devendra Jadhav
wrote:

> Hi,
>
> I have to import data from one database to another, I have to import
> around
> 100000(1Lac) records.
> First I need to check if the record is already imported or not and
> import
> only those records which are not imported.
>
> Here is my logic
>
> $already_imported = get_already_imported_records();
> format of the $already_imported is $already_imported[someid] =
> 'imported';
>
> Now i take all records from another db and iterating through it.
>
> if (!key_exists($already_imported[$new_id])){
> import_function($new_id)
> }else{
> echo 'allready imported'.$already_imported[$new_id];
> }
>
> Now my script is importing same records for more than one time. I am
> not
> able to get through this issue
>
> Is it because of the size of the records or something else...?
>
> Please suggest me some solution which is faster, safe and easy to
> code :D
>
> Thanks in advance
>
> --
> Devendra Jadhav

What are the databases? Both mysql? Or different systems? How do you
define whether the record exists?

Are you running into timeout issues? If so, one trick is to write the
page to process 100 or so records at a time and then use JavaScript to
reload the page to avoid the timeout issues.

Bastien

Sent from my iPod


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php