Sending out large amounts of email

--001636c5b23290290a0464676bfa
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Hi.

Our company is merging with another company and newsletter now needs to go
out to more than 100.000 people. Before it was only a couple of thousands.

I have developed a mail queue using a database and a cronjob, but I am not
in doubt as to what particular solution I need to implement.

I have been running some tests with PHP mail() function, PHPMailer and
PEAR:Mail using 6000 mails at once.

Here's a sumarry of some of the results:

PHP mail() send out 6000 mails in 1.75 seconds.
PHPMailer using PHP mail() send out 6000 mails in 1.87 seconds.
PHPMailer using SMTP send out 6000 mails in 12 seconds (Error without
succes).
PEAR:Mail using PHP mail() send out 6000 mails in > 20 seconds (Apache
reached 100% during this time).

Running several test on PHPMailer using SMTP failed completely with more
than 1000 mails.

Everywhere on the net I read that sending out mail using PHP mail() is slow
and it is a bad idea and that using some mail class with SMTP directly would
be much better. I have tested the run my tests with Postfix as the SMTP.

I have gotten the best results using PHP mail(). When the volume is belove
6000 PHP mail() is by far the fastest.

Would someone mind sharing experience and perhaps shedding some light on
this issue?

What is the best solution in real life dealing with huge amounts of mail?
PHP mail() vs. some class using SMTP or binary sendmail (or wrapper)?

All insights would be appriciated.

Best regards.

Brian

--001636c5b23290290a0464676bfa--

Re: Sending out large amounts of email

> I have been running some tests with PHP mail() function, PHPMailer and
> PEAR:Mail using 6000 mails at once.
>
> Here's a sumarry of some of the results:
>
> PHP mail() send out 6000 mails in 1.75 seconds.
> PHPMailer using PHP mail() send out 6000 mails in 1.87 seconds.
> PHPMailer using SMTP send out 6000 mails in 12 seconds (Error without
> succes).
> PEAR:Mail using PHP mail() send out 6000 mails in > 20 seconds (Apache
> reached 100% during this time).
>
> Running several test on PHPMailer using SMTP failed completely with more
> than 1000 mails.

Probably if you tried to send > 1,000 in one smtp conversation. Postfix
has a setting that will control this (as far as I remember anyway).
You'll need to cycle the connection.

- connect
- send 100
- disconnect

repeat.

> Everywhere on the net I read that sending out mail using PHP mail() is slow
> and it is a bad idea and that using some mail class with SMTP directly would
> be much better. I have tested the run my tests with Postfix as the SMTP.

Depends on the situation. Sometimes using an smtp server is better. Some
hosts only let you send 'x' emails an hour or even 'x' emails every 5
minutes so you have to pause between each email being sent so you don't
flood their servers.

If you're only using your server and your tests show php mail() works,
then use it.

PHPMailer comes in very handy when you have to do one or more of the
following:
- send multipart emails (html + text)
- include attachments
- embed html images in the content
- possibly send to an smtp server

All of that functionality is already built, works, tested etc - you
don't need to do it all again.

> I have gotten the best results using PHP mail(). When the volume is belove
> 6000 PHP mail() is by far the fastest.

What about 10,000?

> What is the best solution in real life dealing with huge amounts of mail?
> PHP mail() vs. some class using SMTP or binary sendmail (or wrapper)?

mail() internally calls the wrapper, I doubt you'd see much of a
difference there.

--
Postgresql & php tutorials
http://www.designmagick.com/


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Sending out large amounts of email

--001636c5aa868652560464682c47
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

>
> What about 10,000?
>

I haven't been able to send out more than 6000 at most. It is proberly
caused by a limit set in Postfix as you wrote.

--001636c5aa868652560464682c47--

Re: Sending out large amounts of email

Brian Hansen wrote:
>
>
> 2009/3/6 Chris >
>
>
> What about 10,000?
>
>
> I haven't been able to send out more than 6000 at most. It is proberly
> caused by a limit set in Postfix as you wrote.

If you're sending emails one by one in a mail() call there is no limit
(one email per recipient)

If you're trying to cc or bcc the email addresses, then there probably
is a limit.

--
Postgresql & php tutorials
http://www.designmagick.com/


--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Sending out large amounts of email

Brian Hansen wrote:
> Hi.
>
> Our company is merging with another company and newsletter now needs to go
> out to more than 100.000 people. Before it was only a couple of thousands.
>
> I have developed a mail queue using a database and a cronjob, but I am not
> in doubt as to what particular solution I need to implement.
>
> I have been running some tests with PHP mail() function, PHPMailer and
> PEAR:Mail using 6000 mails at once.
>
> Here's a sumarry of some of the results:
>
> PHP mail() send out 6000 mails in 1.75 seconds.
> PHPMailer using PHP mail() send out 6000 mails in 1.87 seconds.
> PHPMailer using SMTP send out 6000 mails in 12 seconds (Error without
> succes).
> PEAR:Mail using PHP mail() send out 6000 mails in > 20 seconds (Apache
> reached 100% during this time).
>
> Running several test on PHPMailer using SMTP failed completely with more
> than 1000 mails.
>
> Everywhere on the net I read that sending out mail using PHP mail() is slow
> and it is a bad idea and that using some mail class with SMTP directly would
> be much better. I have tested the run my tests with Postfix as the SMTP.
>
> I have gotten the best results using PHP mail(). When the volume is belove
> 6000 PHP mail() is by far the fastest.
>
> Would someone mind sharing experience and perhaps shedding some light on
> this issue?
>
> What is the best solution in real life dealing with huge amounts of mail?
> PHP mail() vs. some class using SMTP or binary sendmail (or wrapper)?
>
> All insights would be appriciated.
>
> Best regards.
>
> Brian
>

I won't debate the same as everybody else - but you can save yourself
some headaches in 2 simple ways:

1: forget using PHP and just sign up for one of the many professional
bulk mail services who specialise in this [if in doublt count up your
(hours spend * hourly rate - cost of service) and thats how much you'll
save - let alone headaches

2: if your like me and want to just do everything for the sake of
learning then send each email to chunks of 100+ or so people by adding
them all in the bcc header field; then every single call to "mail()"
will be 100x more productive

side notes:
off-list I'm going to send you a class i made a couple of years ago
which popen's sendmail via the command line and sends through a raw mime
encoded email to multiple recipients extremely quickly and without error
- was the best implementation I could get after much testing and playing
for weeks - not saying you should use it, but it is packed with comments
and notes you may find useful - linux only uses sendmail

note 2: remember you're spf!

regards,

nathan

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Sending out large amounts of email

On Thu, Mar 5, 2009 at 6:20 PM, Brian Hansen wrote:
> Hi.
>
> Our company is merging with another company and newsletter now needs to go
> out to more than 100.000 people. Before it was only a couple of thousands.
>
> I have developed a mail queue using a database and a cronjob, but I am not
> in doubt as to what particular solution I need to implement.
>
> I have been running some tests with PHP mail() function, PHPMailer and
> PEAR:Mail using 6000 mails at once.
>
> Here's a sumarry of some of the results:
>
> PHP mail() send out 6000 mails in 1.75 seconds.
> PHPMailer using PHP mail() send out 6000 mails in 1.87 seconds.
> PHPMailer using SMTP send out 6000 mails in 12 seconds (Error without
> succes).
> PEAR:Mail using PHP mail() send out 6000 mails in > 20 seconds (Apache
> reached 100% during this time).
>
> Running several test on PHPMailer using SMTP failed completely with more
> than 1000 mails.
>
> Everywhere on the net I read that sending out mail using PHP mail() is slow
> and it is a bad idea and that using some mail class with SMTP directly would
> be much better. I have tested the run my tests with Postfix as the SMTP.
>
> I have gotten the best results using PHP mail(). When the volume is belove
> 6000 PHP mail() is by far the fastest.
>
> Would someone mind sharing experience and perhaps shedding some light on
> this issue?
>
> What is the best solution in real life dealing with huge amounts of mail?
> PHP mail() vs. some class using SMTP or binary sendmail (or wrapper)?
>
> All insights would be appriciated.
>
> Best regards.
>
> Brian
>

I've done a lot of work trying to get mail servers/php code to work
reliably together in sending bulk mail. It's not trivial business. If
you have control of the setup, change the mail server to just queue
your mail instead of sending it directly. I did this and it solved a
lot of really nasty headaches. I use Zend_Mail & this method to send
custom generated html emails to a lot of people and it never fails.

It might not work for you though, so good luck!

--
http://www.voom.me | EFnet: #voom

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Sending out large amounts of email

Hello,

on 03/05/2009 08:20 PM Brian Hansen said the following:
> Everywhere on the net I read that sending out mail using PHP mail() is slow
> and it is a bad idea and that using some mail class with SMTP directly would
> be much better. I have tested the run my tests with Postfix as the SMTP.

This is a totally wrong misconception. Using SMTP is the slowest method
to queue messages.

The misconception comes from the fact that sendmail is used in many
servers and by default sendmail attempts to send the message right away
when you queue it with the mail function.

However sendmail has options to just drop the message in the local queue
the message instead of deliverying it immediately.

SMTP queueing is much slower because it uses a TCP (SMTP) connection to
connect to the local server and drop the message queue, while when you
use the regular mail function it just drops the message in the queue
without TCP overhead.

Your tests demonstrated that mail() outperforms SMTP because either you
are not using sendmail or it is sendmail with the imediate delivery
option disabled.

You may want to take a look at the slides of a presentation that
addressed this issue here.

If you jump to slide 13 you can see a diagram that shows the path of the
messages when you send them via different methods.

As you may notice using SMTP is the long way to make the message reach
the local mail server queue.

http://www.phpclasses.org/browse/video/3/package/9.html


> I have gotten the best results using PHP mail(). When the volume is belove
> 6000 PHP mail() is by far the fastest.
>
> Would someone mind sharing experience and perhaps shedding some light on
> this issue?
>
> What is the best solution in real life dealing with huge amounts of mail?
> PHP mail() vs. some class using SMTP or binary sendmail (or wrapper)?

The fastest way is to just drop the messages in the pickup folder that
some mail servers offer. I am not sure which Linux/Unix servers support
that, maybe postfix does, but I am sure Microsoft Exchange supports it.

Anyway, if you use the regular mail function it may be fine too.

Personally I use qmail but I can control the mail server I have in the
machine of my sites, and just call qmail-inject program directly instead
of the regular sendmail wrapper that the mail function calls.

Other than that, queueing the messages is just one step of bulk message
delivery. Often you need to compose and send complex messages that have
characters then need to be encoded, maybe attachments, embeded pictures
in HTML, etc.. That adds quite a bit of overhead to PHP script message
delivery.

I use the MIME message class below that the presentation above refers.
It provides a bag of tricks to optimize bulk deliveries, depending on
whether you need to personalize or not for each recipient. The talk
about the slides above also address those issues and how they can be
solved to maximize delivery performance and reliability with that class:

http://www.phpclasses.org/mimemessage



--

Regards,
Manuel Lemos

Find and post PHP jobs
http://www.phpclasses.org/jobs/

PHP Classes - Free ready to use OOP components written in PHP
http://www.phpclasses.org/

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Sending out large amounts of email

Hi,

> Our company is merging with another company and newsletter now needs to go
> out to more than 100.000 people.

Out source it. It will cost you far less in the long run. And the
short run. I tried Jango mail and they seemed OK, though I didn't have
to talk to their support, and my use was very brief. These types of
company will also have far more success in getting the emails
delivered (and not flagged as spam).

--
Richard Heyes

HTML5 Canvas graphing for Firefox, Chrome, Opera and Safari:
http://www.rgraph.net (Updated February 28th)

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

RE: Sending out large amounts of email

From: Chris
> Brian Hansen wrote:
>> 2009/3/6 Chris >
>> =20
>>=20
>> What about 10,000?
>>=20
>>=20
>> I haven't been able to send out more than 6000 at most. It is
proberly=20
>> caused by a limit set in Postfix as you wrote.
>=20
> If you're sending emails one by one in a mail() call there is no limit

> (one email per recipient)
>=20
> If you're trying to cc or bcc the email addresses, then there probably

> is a limit.

You should also talk to your ISP and email admins to find out how to
avoid ending up on a SPAM blacklist after sending out that many emails.
You're likely to trigger more than one automatic listing with that many
messages.

Bob McConnell

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Sending out large amounts of email

--001636c5bdc4e7ca6e04647ab4cf
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Thank you all for some great answer and insights.

Special thanks to Chris, Nathan and Manuel for very valuable pointers and
sharing of information and experience.

For those interested in the subject..

We have our own servers, and since we want to be in control of the situation
ourselves we are not going to out-source the task.

I have chosen to go with a setup using PHPMailer on a mailserver running
Postfix and PostgreSQL. I have come to the conclusion that running PHPMailer
through the Sendmail binary (Postfix wrapper in this case) is the fastest
method.

Using Postfix this way doesn't use a queue, so what I do is that the script
saves the body and headers in a database table. In another table alle the
different email addresses is saved using a loop. Currently I am looking into
a better approach, but I haven't had the time yet. But I was thinking of
something like having a field for marking customers as receivers.

A cronjob is then run each two minuts (or something like that) calling a
small PHP script that uses PHPMailer to send out about 1000-3000 mails at a
time (it takes about 1 sec. and only about 24% of CPU power in our case
during that second). It loops through the customer emails.

Thanks to Chris I have looked into the subject of SPF, and I have created
the nesecary SPF TXT records for the relevant domain (thanks a thousand
Chris, I had completely forgotten about SPF).

I have run a couple of tests and so far every mailboks I have testet (Gmail,
Hotmail, different ISP's etc) have received the mail directly in the inbox,
except for Hotmail (typical) that puts the mail in the Junk box, but atleast
it gets through!

I have found this solution rather simple and very easy to maintain.

Best regards.

2009/3/6 Brian Hansen

> Hi.
>
> Our company is merging with another company and newsletter now needs to go
> out to more than 100.000 people. Before it was only a couple of thousands.
>
> I have developed a mail queue using a database and a cronjob, but I am not
> in doubt as to what particular solution I need to implement.
>
> I have been running some tests with PHP mail() function, PHPMailer and
> PEAR:Mail using 6000 mails at once.
>
> Here's a sumarry of some of the results:
>
> PHP mail() send out 6000 mails in 1.75 seconds.
> PHPMailer using PHP mail() send out 6000 mails in 1.87 seconds.
> PHPMailer using SMTP send out 6000 mails in 12 seconds (Error without
> succes).
> PEAR:Mail using PHP mail() send out 6000 mails in > 20 seconds (Apache
> reached 100% during this time).
>
> Running several test on PHPMailer using SMTP failed completely with more
> than 1000 mails.
>
> Everywhere on the net I read that sending out mail using PHP mail() is slow
> and it is a bad idea and that using some mail class with SMTP directly would
> be much better. I have tested the run my tests with Postfix as the SMTP.
>
> I have gotten the best results using PHP mail(). When the volume is belove
> 6000 PHP mail() is by far the fastest.
>
> Would someone mind sharing experience and perhaps shedding some light on
> this issue?
>
> What is the best solution in real life dealing with huge amounts of mail?
> PHP mail() vs. some class using SMTP or binary sendmail (or wrapper)?
>
> All insights would be appriciated.
>
> Best regards.
>
> Brian
>
>
>

--001636c5bdc4e7ca6e04647ab4cf--

Re: Re: Sending out large amounts of email

> thanks a thousand

A thousand? That's a bit stingy - usually it's "thanks a million"...

:-)

--
Richard Heyes

HTML5 Canvas graphing for Firefox, Chrome, Opera and Safari:
http://www.rgraph.net (Updated February 28th)

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: Re: Sending out large amounts of email

--001636e0aedb19a2cc046484c1d6
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On Sat, Mar 7, 2009 at 4:11 PM, Richard Heyes wrote:

> > thanks a thousand
>
> A thousand? That's a bit stingy - usually it's "thanks a million"...


In this particular case I guess thousand is a larger value than million :D

takes a longer to type

>
>
> :-)
>
> --
> Richard Heyes
>
> HTML5 Canvas graphing for Firefox, Chrome, Opera and Safari:
> http://www.rgraph.net (Updated February 28th)
>
> --
> PHP General Mailing List (http://www.php.net/)
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

--001636e0aedb19a2cc046484c1d6--