PHPExcel with large files (27,000+ rows)

--005045016763dfe3800491cede2f
Content-Type: text/plain; charset=ISO-8859-1

I'm currently working on a project that requires the parsing of excel files.
Basically the user uploads an excel file, and then a script needs to save a
row in a Postgres database for each row in the excel file. The issue we are
having is that when we task PHPExcel with parsing an excel file with, say
27k rows, it explodes with a memory error. I've read up on the PHPExcel
forums and we've tried cell caching as well as ReadDataOnly, they do not
seem to be sufficient.

Does anyone here know of a way to do this? Surely there is a way to parse a
large excel file with PHP. This is also NOT an on-demand service. That is,
when someone uploads a file they get a task_id which allows them to check
the status of their excel file. So the solution does not need to be a fast
one!


Thanks,
Chris.

--005045016763dfe3800491cede2f--

Re: PHPExcel with large files (27,000+ rows)

chris h wrote:

> I'm currently working on a project that requires the parsing of excel=

> files. Basically the user uploads an excel file, and then a script
> needs to save a row in a Postgres database for each row in the excel
> file. The issue we are having is that when we task PHPExcel with
> parsing an excel file with, say 27k rows, it explodes with a memory
> error. I've read up on the PHPExcel forums and we've tried cell
> caching as well as ReadDataOnly, they do not seem to be sufficient.
>=20
> Does anyone here know of a way to do this? Surely there is a way to
> parse a large excel file with PHP. =20

If your excel file is or can be transformed to XML, I would just use
XSLT. No PHP needed.=20



--=20
Per Jessen, Zürich (19.1°C)


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: PHPExcel with large files (27,000+ rows)

I use this: http://code.google.com/p/php-csv-parser/

No idea if it's any better than your current solution. I presume
you've tried extending PHP's memory limit?

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: PHPExcel with large files (27,000+ rows)

On Tue, Oct 5, 2010 at 12:39 AM, chris h wrote:
> I'm currently working on a project that requires the parsing of excel fil=
es.
>  Basically the user uploads an excel file, and then a script needs t=
o save a
> row in a Postgres database for each row in the excel file.  The issu=
e we are
> having is that when we task PHPExcel with parsing an excel file with, say
> 27k rows, it explodes with a memory error.  I've read up on the PHPE=
xcel
> forums and we've tried cell caching as well as ReadDataOnly, they do not
> seem to be sufficient.
>
> Does anyone here know of a way to do this? Surely there is a way to parse=
a
> large excel file with PHP.  This is also NOT an on-demand service. =
 That is,
> when someone uploads a file they get a task_id which allows them to check
> the status of their excel file.  So the solution does not need to be=
a fast
> one!
>
>
> Thanks,
> Chris.
>

1. Remove any variable that contains big object if its not necessary.
2. Use unset when applicable
3. Read chunk by chunk.
4. Profile it to find the exact place where you are wasting memory.
Optimizing that little portion of code can improve memory performance.

--=20
Shiplu Mokadd.im
My talks, http://talk.cmyweb.net
Follow me, http://twitter.com/shiplu
SUST Programmers, http://groups.google.com/group/p2psust
Innovation distinguishes bet ... ... (ask Steve Jobs the rest)

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: PHPExcel with large files (27,000+ rows)

--0016e6d369487a37870491cf4751
Content-Type: text/plain; charset=ISO-8859-1

Thanks Jessen/Marc, though the user provided format can be in xls, xlsx, or
csv. So i need a solution to support all formats.

Thanks for the ideas shiplu I'll get with the team and see if there's
anything there we aren't trying.


Chris.


On Mon, Oct 4, 2010 at 3:01 PM, shiplu wrote:

> On Tue, Oct 5, 2010 at 12:39 AM, chris h wrote:
> > I'm currently working on a project that requires the parsing of excel
> files.
> > Basically the user uploads an excel file, and then a script needs to
> save a
> > row in a Postgres database for each row in the excel file. The issue we
> are
> > having is that when we task PHPExcel with parsing an excel file with, say
> > 27k rows, it explodes with a memory error. I've read up on the PHPExcel
> > forums and we've tried cell caching as well as ReadDataOnly, they do not
> > seem to be sufficient.
> >
> > Does anyone here know of a way to do this? Surely there is a way to parse
> a
> > large excel file with PHP. This is also NOT an on-demand service. That
> is,
> > when someone uploads a file they get a task_id which allows them to check
> > the status of their excel file. So the solution does not need to be a
> fast
> > one!
> >
> >
> > Thanks,
> > Chris.
> >
>
> 1. Remove any variable that contains big object if its not necessary.
> 2. Use unset when applicable
> 3. Read chunk by chunk.
> 4. Profile it to find the exact place where you are wasting memory.
> Optimizing that little portion of code can improve memory performance.
>
> --
> Shiplu Mokadd.im
> My talks, http://talk.cmyweb.net
> Follow me, http://twitter.com/shiplu
> SUST Programmers, http://groups.google.com/group/p2psust
> Innovation distinguishes bet ... ... (ask Steve Jobs the rest)
>

--0016e6d369487a37870491cf4751--